What we can learn about historical fiction reading (and reviewing) from an AI chatbot

Over the past several weeks, I’ve been fascinated by ChatGPT, the artificial intelligence chatbot that responds to prompts in a conversational style which mimics human writing. Educators have raised concerns about students using it to plagiarize essays; technical writers (and others) are worried about an AI agent putting them out of work. The applications and ethical ramifications of this program are just beginning to be studied.

When it comes to requests for basic information on a topic, ChatGPT returns text that's logically organized, easily readable, and mostly accurate (it does make mistakes). It’s also scary fast. When I asked it to write a sonnet about historical fiction, the results were impressive but flawed. People on the Historical Novel Society’s Facebook group had an interesting mix of responses to it.

I wanted to explore the output created by ChatGPT so I could see its strengths and defects firsthand – and so I could better recognize AI-generated writing when I came across it in other circumstances. (There are new programs out there to detect whether content was written by a robot.) ChatGPT isn’t going away and will only improve over time, so it’s to our benefit to become familiar with its capabilities. So I prompted it with some requests to see what could be gleaned from the experience.

How good is ChatGPT at writing book reviews, I wondered. The system’s knowledge base ends with 2021, so this exercise won’t work for new releases, but I was curious what information it knew about older titles, and about historical fiction in general.

First, I asked it to write a sample book review for a novel about pilots during WWII (I didn’t specify a particular book). The result was:

Example 1:

Flyboys: A Novel of WWII by John Smith is a powerful and moving historical novel set during World War II. The story follows a group of young pilots as they train to become fighter pilots and then head off to fight in the Pacific Theater.

One of the strengths of this novel is the way it brings to life the personal stories of the pilots. Through their eyes, we see the horrors of war and the sacrifices they made for their country. Smith does an excellent job of capturing the camaraderie and sense of brotherhood that developed among the pilots, as well as the fear and uncertainty they faced on a daily basis.

The novel also explores the moral and ethical dilemmas that the pilots faced, including the question of whether or not to shoot down enemy pilots who have bailed out of their planes. The characters are well-developed and feel authentic to the time period. The main protagonist, a young pilot named Jack, is a relatable and sympathetic character who the reader can't help but root for.

The novel is also historically accurate and provides a great deal of information about the air battles that took place during World War II, which makes it a great novel for readers who are interested in the history of WWII and the role of pilots in the war.

The writing is engaging and evocative, and the pacing of the novel keeps the reader on the edge of their seat. The ending is bittersweet, but it leaves the reader with a sense of the human cost of war.

Overall, "Flyboys: A Novel of WWII" is a must-read for fans of historical fiction and World War II enthusiasts. It is well-written, well-researched, and filled with relatable characters and a compelling storyline that will keep readers engaged until the very end. It is a highly recommended novel for anyone interested in the history of WWII, the role of pilots, and the human cost of war.

--

Does this sound like a novel you’d want to read? Or maybe you’ve already read something very close to it?

Since I gave it a generic request, it provided a generic review. Still, it picked up on a basic plotline for a book with this focus, reflected these common elements within the review (the fact that many are stories of camaraderie and coming-of-age journeys for their male protagonist), and provided commentary on positive qualities readers expect to find in historical fiction. That is: well-developed and authentic characters, historical accuracy, a moving and powerful storyline, and interesting background details.

Next – several days later – I asked ChatGPT for a review on a novel about a young woman growing up in medieval England. The result named a specific book, but the plotline doesn't fit that book, aside from getting the era right.

Example 2:

The Rose of York: Love and War is a captivating historical novel set in medieval England during the Wars of the Roses. The story follows the life of a young woman named Rose as she navigates the complexities of medieval society, from her childhood in a small village to her eventual rise as a powerful and influential woman.

One of the strengths of this novel is the way it brings to life the rich and complex world of medieval England. The author does an excellent job of capturing the political and social landscape of the time, from the power struggles between the different noble houses to the everyday lives of the common people. The novel also explores the role of women in medieval society and how Rose's determination and strength of character allows her to transcend the limitations placed on her as a woman.

The characters are well-developed and feel authentic to the time period. The main protagonist, Rose, is a strong and relatable character who the reader can't help but root for. The love story that develops throughout the novel is also engaging and adds an emotional depth to the story.

The writing is evocative, and the pacing of the novel keeps the reader engaged throughout. The ending is satisfying and leaves the reader with a sense of the human cost of war and the strength of the human spirit.

Overall, "The Rose of York: Love and War" is a must-read for fans of historical fiction. It is well-written, well-researched, and filled with richly drawn characters and a compelling storyline that will keep readers engaged until the very end. It is a highly recommended novel for anyone interested in the Middle Ages, the Wars of the Roses, and the role of women in history.

--

This second review has elements worth noting. It draws on tropes for women’s fiction set in medieval times (the heroine’s personal growth journey, how she reacts to the restrictions placed on women in past eras, and the fact that there’s a love story – not mentioned at all in the sample review about WWII pilots). Taken together with the first review, we see how these AI-generated reviews are structured. The underlined sections show the text repeated in both reviews. It seems there’s an underlying template that ChatGPT is following, while filling in the blanks with material appropriate to the subject and era.

Thirdly, I requested a critical review of a WWII novel. (I initially asked for a snarky review, just for fun, but it tsked at me and told me it wouldn’t do that.)

Example 3:

The War Diaries: A Novel of WWII by John Doe is a historical novel set during the Second World War. The story follows a group of soldiers as they fight their way through Europe, and the struggles they face both on and off the battlefield.

One of the weaknesses of this novel is that it lacks historical accuracy. There are several inaccuracies in the descriptions of battles and settings, which undermines the credibility of the story. Additionally, the characters are not well-developed, and their actions and motivations are not always clear.

The author's writing style is also problematic. The dialogue is often clunky and unrealistic, and the author's attempts at humor fall flat. The pacing of the novel is also uneven, with long stretches of slow-moving action interspersed with brief bursts of action.

Overall, "The War Diaries: A Novel of WWII" falls short in providing a credible and engaging story. The lack of historical accuracy and the poor writing style make it hard for the reader to fully immerse in the story. It's not a bad book, but it's not a must-read for fans of historical fiction set during WWII, and readers might want to look for other options.

--

Again, we can learn more about reader expectations from this review of a fictional novel. (ChatGPT made up the title and author.) It’s a coherently organized review that shows multiple reasons why a historical novel can fail to resonate with readers.

Lastly, I got out of generic mode and asked: Can you write a book review for All the Light We Cannot See? Surely it would do better with a highly acclaimed, bestselling historical novel.

Example 4:

"All The Light We Cannot See" is a historical fiction novel written by Anthony Doerr. The novel takes place in Europe during World War II and follows the parallel stories of two main characters: Marie-Laure, a blind French girl, and Werner, a German boy. The story is beautifully written, with a powerful and evocative prose that draws the reader into the characters' lives and experiences.

The novel is about the impact of war on the lives of ordinary people, and how it can change them. The author does an excellent job of capturing the fear, uncertainty, and hope that people felt during the war. Doerr's descriptions of the war-torn cities and the devastation caused by bombing are particularly powerful, and they help to convey the scale of the destruction and suffering caused by the war.

The author also explores themes of love, family, and sacrifice, with the characters learning to overcome their fears and biases. The relationship between Marie-Laure and Werner is particularly well-done, as the author explores the different paths their lives take, and how they are ultimately brought together.

The author's writing style is evocative, poetic and he is able to create a powerful atmosphere, it is a book that is not only informative but also emotionally engaging. The book is a beautifully written, thought-provoking novel that is sure to stay with the reader long after they have finished reading it.

Overall, "All the Light We Cannot See" is a must-read for fans of historical fiction, war literature, and for anyone who is interested in exploring the human experience during one of the most devastating conflicts in human history.

--

So what do you think?

On the plus side, it’s structured like a standard book review, and if you’ve read the novel (and I have), you’ll notice it gets some things right. It’s mostly grammatically correct, too.

But as a review of Doerr’s novel, which has been analyzed in detail in many corners of the web, it’s bland and superficial. Not only does it lack style and artistry – typical in AI-generated text – but there’s almost no information about the characters, plot, or setting. When I read reviews, I appreciate perceptive examples of what worked well, what didn’t, and why, preferably with originality of expression. This review reads like someone ran out of time and was phoning it in, or that they briefly glanced at the publisher’s blurb but didn’t actually read the book. Also, despite its emphasis on themes, it doesn’t really capture what All the Light We Cannot See is about.

I’ve repeated this same prompt multiple times, with only a slight variation in the AI responses.

In writing reviews, I know how tempting it can be to fall back on clichés and familiar phrases. Everyone uses them to some degree, and they can pass unnoticed if the reviewer points out specific examples and demonstrates original insight. The reviews written by the chatbot, especially in the fourth entry, focus on historical fiction tropes while omitting details on an author’s unique accomplishments. Would any of the AI-generated reviews convince you to read the book described (if it existed)?

In the end – and thanks for reading this lengthy post – we can learn numerous things from these AI-written reviews, but the takeaways have more to do with genre and reviewing conventions than the books themselves. I welcome additional thoughts, as well as details on your own explorations with this new tool.