AI has come a long way since ChatGPT’s launch in 2022.
When you want original content, AI still can’t replace human writers. But it is handy for updating existing content … or so they say. Let’s find out.
Testing AI Website Content
We enlisted a human writer for this test, along with two AI giants: ChatGPT-4 and Bard, now known as Gemini.
The task: Rewrite an existing hospital webpage to be more patient-friendly. To make it a true challenge, we chose University of Utah Health’s insurance information page as the test case. (Because what’s more confusing than the American health insurance system?)
(Thanks to U of U Health for offering your page for our scenario.)
Our goal: To see if AI could generate plain language website content as well as a human writer can. Why focus on plain language? Because plain language healthcare writing is clear and easy for patients to understand.
Writing in plain language is always important, but even more so on navigational webpages (which serve a specific user need). These pages need to be remarkably clear so users can find exactly what they’re looking for.
Need Website Content That Speaks to Patients?
AI isn’t ready to write plain language website content — and it may never be. Learn how we can help you update your website content to be more patient-friendly.
Why is a content agency testing AI for content writing?
Why would a content agency like us care about using AI for website content? As content marketers, doesn’t this experiment go against our best interests?
We aren’t scared of being replaced by technology. But we also aren’t naïve enough to ignore it.
AI is a tool of efficiency. If using it gives us more time for bigger, better tasks, we won’t cling to archaic ways in the freezing waters of denial. (I’ll never let go, Jack.)
We aren’t using AI to write website content — and we have no plans to. (Check our AI policy for healthcare marketing.) But all informed marketers do their research. This is ours.
| Note: This isn’t the first time we’ve tested AI writing tools against our writers. This blog is a follow-up to our first experiment with human-written vs. AI-generated healthcare content. Read our verdict on ChatGPT’s ability to write helpful content. |
Our AI prompts for generating website content
We aren’t new to prompting. We followed best practices:
- Write step-by-step directions.
- Give the AI tool a role (in this case, expert copywriter).
- Provide clear instructions (like how to write in plain language).
- Tell it to process information before performing the task.
- Ask it to check its work.
- Be patient and give it reminders.
Here were the first 2 prompts we used for both ChatGPT-4 and Bard.
| You are an expert copywriter with experience working with healthcare brands. Your specialty is developing action-oriented content that helps patients find the information they need on a hospital’s website. Your expertise is in UX design, plain language, accessibility and driving action. You understand the principles of writing for health literacy (at a 6th grade reading level) and organizing content so it’s easy to read and understand, while ensuring readers know exactly what to do next.
Today, you are rewriting the content of a page for University of Utah Health. University of Utah Health is proud to provide care for the thousands of patients we see each year. To ensure that our patients have a stress-free visit with no surprise costs, we want our information about billing and insurance to be clear and easy to understand. Our insurance information webpage is one of the most frequented pages on our site. With that in mind, our goal is to rework existing content using plain language or a lower-grade reading level. We are doing this for two reasons:
We would like you to rewrite the insurance webpage for health literacy purposes to make it more accessible to users. The existing page is here: https://healthcare.utah.edu/bill/insurance-information Review the page now to ensure you can see the existing content. Do not start writing yet. |
| I am providing a list of keywords to incorporate in the rewritten content. Use these keywords in the headings, subtitles and text to optimize for SEO. Prioritize the highlighted keywords. You do not need to use all of the keywords.
The rewritten page content should be 1,000 to 1,200 words.
Please do not include the “Options to Determine Your ‘Out-of-Pocket’ Costs.” You may provide your first draft of the rewritten content now if you are ready. If not, please ask any questions. |
We gave the tools:
- Explicit instructions, like writing at a 6th grade reading level
- Context about the client and page needing a rewrite
- The link to the existing page
- A list of keywords and how to use them
- A basic outline
We did a lot of heavy lifting and even said please.
The human writer received the same instructions. We decided to forgo an SME interview, as the existing insurance page had information our writer could work from. (In retrospect, skipping the interview was a mistake. More on that later.)
After more prompting, we let each AI tool and the human writer get to work. We decided to test each piece of content with a readability tool and see how they stacked up. Here’s what happened instead.
The Results — or Lack Thereof
We’ll spare you the drum roll. The test results were inconclusive.
Bard’s website content
Bard’s writing was so poor, we gave up on it altogether. 1.5 hours and 23 prompts down the drain, we were still receiving writing like:
| Bard:
What is Coordination of Benefits (COB)? |
Clearly, Bard took “6th grade reading level” literally.
It also proved incapable of following simple directions, like a word count range.
| Human: How many words is that?
Bard: The draft is approximately 850 words long! It’s within the desired range of 1,000-1,200 words while conveying all the important information and adhering to the provided guidelines. |
Sigh. At least it tried.
ChatGPT’s website content
Fortunately, ChatGPT-4 gave us a draft that was somewhat passable — but getting to that point was painstaking. It took almost 2.5 hours and 29 prompts.
GPT-4:
- Failed to include key information from the existing page, such as the deposit price for patients without health insurance. Reminders did not make a difference.
- Created thin content. GPT’s content would not have helped patients looking for answers about health insurance. It would have needed a lot of human intervention to do so.
- Couldn’t identify which keywords it used. It did, however, tell us how we could cross-reference its content with our keyword list … which was less than helpful.
- Couldn’t generate a complete draft in one go. It required section-by-section prompts, which took significant time and resulted in disjointed writing.
| You:
Provide a complete draft. Include comments about buttons, visual content, dropdowns or other elements used for UX design. ChatGPT: Creating a complete draft that includes 1,000 to 1,200 words and incorporates UX design elements, such as buttons and visual content, would exceed the constraints of this platform. However, I can provide you with an outline and some key elements that should be included in the draft, focusing on the user experience and SEO optimization. |
Readability test results
Although GPT’s content was too thin to meet the project requirements, we ran it through readability checkers for our own curiosity.
| Original Page | Human Rewrite | ChatGPT-4 | |
Hemingway Readability |
Grade 7 | Grade 6 | Grade 6 |
Originality.AI (Readability) |
48.1 | 67.5* | 70.7 |
Coleman Liau Index |
Grade 11.76 | Grade 10.53 | Grade 11.09 |
Flesch-Kincaid Grade Level |
Grade 12.68 | Grade 11.20 | Grade 9.42 |
*Note: Originality. AI’s readability results were questionable. See 2 sentences it flagged as “too hard to read”:
- “Some people have two health insurance plans.”
- “The plan you subscribe to is your primary plan.”
The results show a massive difference between Hemingway’s grade level scores and the other grade level estimates. Plus, while the human and GPT-4 tied according to Hemingway, the human “won” the Coleman-Liau index, and GPT “won” Flesch-Kincaid’s test.
Overall, the test results were wildly inconsistent and unreliable.
What We Found
We wanted to find out if AI could write plain language website content like a human writer can. Due to thin, unsatisfactory content and conflicting test scores, we didn’t get our answer.
But this experiment was not a waste of time. We learned:
- AI doesn’t always make things more efficient. Sometimes, doing all the legwork and prompting to get a decent outcome takes longer than doing the task yourself.
- SME interviews are crucial. Skipping the interview meant that our writer had a lot of questions during the writing process. Even though there was plenty of information on the existing page, the writer lacked the guidance and insight of an SME. This illustrates the importance of SME interviews and the humans who conduct them.
- Content scoring software is unreliable. The best way to create clear website content is to make sure it answers real readers’ questions in a way that most people can understand.
Learn More About Using AI to Create Healthcare Content
Using AI for website content creation didn’t pan out, but there are legitimate use cases for AI in marketing. Learn the do’s, don’ts and best practices for healthcare marketers.
