Case Study: Can AI Write Website Content in Plain Language?

AI has come a long way since ChatGPT’s launch in 2022.

When you want original content, AI still can’t replace human writers. But it is handy for updating existing content … or so they say. Let’s find out.

Testing AI website content

We enlisted a human writer for this test, along with two AI giants: ChatGPT-4 and Gemini (formerly known as Bard).

The task: Rewrite an existing hospital webpage to be more patient-friendly. To make it a true challenge, we chose University of Utah Health’s insurance information page as the test case. (Because what’s more confusing than the American health insurance system?)

(Thanks to U of U Health for offering your page for our scenario.)

Our goal: To see if AI could generate plain language website content as well as a human writer can. Why focus on plain language? Because plain language healthcare writing is clear and easy for patients to understand.

Writing in plain language is always important, but even more so on navigational webpages (which serve a specific user need). These pages need to be remarkably clear so users can find exactly what they’re looking for.

Why is a content agency testing AI for content writing?

Why would a content agency like us care about using AI for website content? As content marketers, doesn’t this experiment go against our best interests?

We aren’t scared of being replaced by technology. But we also aren’t naïve enough to ignore it.

AI is a tool of efficiency. If using it gives us more time for bigger, better tasks, we won’t cling to archaic ways in the freezing waters of denial. (I’ll never let go, Jack.)

We aren’t using AI to write website content — and we have no plans to. (Check our AI policy for healthcare marketing.) But all informed marketers do their research. This is ours.

Our AI prompts for generating website content

We aren’t new to prompting. We followed best practices:

• Write step-by-step directions.
• Give the AI tool a role (in this case, expert copywriter).
• Provide clear instructions (like how to write in plain language).
• Tell it to process information before performing the task.
• Ask it to check its work.
• Be patient and give it reminders.

Here were the first 2 prompts we used for both ChatGPT-4 and Gemini.

We gave the tools:

• Explicit instructions, like writing at a 6^th grade reading level
• Context about the client and page needing a rewrite
• The link to the existing page
• A list of keywords and how to use them
• A basic outline

We did a lot of heavy lifting and even said please.

The human writer received the same instructions. We decided to forgo an SME interview, as the existing insurance page had information our writer could work from. (In retrospect, skipping the interview was a mistake. More on that later.)

After more prompting, we let each AI tool and the human writer get to work. We decided to test each piece of content with a readability tool and see how they stacked up. Here’s what happened instead.

The results — or lack thereof

We’ll spare you the drum roll. The test results were inconclusive.

Gemini’s website content

Gemini’s writing was so poor, we gave up on it altogether. 1.5 hours and 23 prompts down the drain, we were still receiving writing like:

Clearly, Gemini took “6^th grade reading level” literally.

It also proved incapable of following simple directions, like a word count range.

Sigh. At least it tried.

ChatGPT’s website content

Fortunately, ChatGPT-4 gave us a draft that was somewhat passable — but getting to that point was painstaking. It took almost 2.5 hours and 29 prompts.

GPT-4:

• Failed to include key information from the existing page, such as the deposit price for patients without health insurance. Reminders did not make a difference.
• Created thin content. GPT’s content would not have helped patients looking for answers about health insurance. It would have needed a lot of human intervention to do so.
• Couldn’t identify which keywords it used. It did, however, tell us how we could cross-reference its content with our keyword list … which was less than helpful.
• Couldn’t generate a complete draft in one go. It required section-by-section prompts, which took significant time and resulted in disjointed writing.

Readability test results

Although GPT’s content was too thin to meet the project requirements, we ran it through readability checkers for our own curiosity.

*Note: Originality. AI’s readability results were questionable. See 2 sentences it flagged as “too hard to read”:

• “Some people have two health insurance plans.”
• “The plan you subscribe to is your primary plan.”

The results show a massive difference between Hemingway’s grade level scores and the other grade level estimates. Plus, while the human and GPT-4 tied according to Hemingway, the human “won” the Coleman-Liau index, and GPT “won” Flesch-Kincaid’s test.

Overall, the test results were wildly inconsistent and unreliable.

What we found

We wanted to find out if AI could write plain language website content like a human writer can. Due to thin, unsatisfactory content and conflicting test scores, we didn’t get our answer.

But this experiment was not a waste of time. We learned:

1. AI doesn’t always make things more efficient. Sometimes, doing all the legwork and prompting to get a decent outcome takes longer than doing the task yourself.
2. SME interviews are crucial. Skipping the interview meant that our writer had a lot of questions during the writing process. Even though there was plenty of information on the existing page, the writer lacked the guidance and insight of an SME. This illustrates the importance of SME interviews and the humans who conduct them.
3. Content scoring software is unreliable. The best way to create clear website content is to make sure it answers real readers’ questions in a way that most people can understand.