Before we begin, let’s make one thing clear — we’ve always stood behind human-first, human-written content at Aha Media Group. We know our biases as a premium content agency. And that’s exactly why we decided to test them.
In the wake of Google’s lightened stance on AI-generated healthcare content (read the latest guidelines on helpful content) we ran an experiment.
We wanted to see how ChatGPT-3 and ChatGPT-4 healthcare content compared to human-written content. So we put them to the test. First, ChatGPT-3 vs. 4, then the winning piece vs. a human writer’s content.
Here’s how we set it up.
| Note: We tested AI writing tools against our writers again in our Case Study: Can AI Write Website Content in Plain Language? Read our verdict on AI’s ability to write website content using plain language. |
About the Experiment
To keep things fair, we gave ChatGPT-3, ChatGPT-4 and our writer the same assignment: Write a webpage for the University of Utah Health Women’s Health Services. We looped in UHealth for feedback on all 3 pieces.
The human-written content went through our usual process. Our writer interviewed a subject matter expert (SME) at UHealth and worked with our editor to send a polished piece.
ChatGPT Healthcare Prompts
For the AI-generated content, we used prompts with specific instructions for both GPT-3 and GPT-4. Here was our first prompt:
A series of follow-up prompts shared the information we include in content briefs for writers:
- Specifics about the webpage, including content goals
- Custom details about the client and their services/program
- The primary and related keywords
- UHealth voice and tone guidelines
We also gave GPT-3 and GPT-4 detailed instructions that we’d expect writers to know:
- Web writing techniques, including subtitle usage, plain language and active vs. passive voice
- An example of how to write in second person
- Directions to use adjectives and adverbs sparingly
- A basic outline for the page, including subtopics
See the outline:
Our last ChatGPT healthcare prompt asked GPT-3 and GPT-4 to suggest internal links within the page to other UHealth pages.
The Review Process
Aside from proofreading, we sent the AI healthcare content to UHealth without human intervention. UHealth then reviewed and edited. The GPT-3 piece took 56 minutes to edit and GPT-4 took 32 minutes. UHealth shared feedback on both pieces and chose the best one to compare with our writer’s.
Then, UHealth ran the winning ChatGPT healthcare content and the human writer’s piece through their usual review process with a doctor who didn’t know which was which. The doctor fact-checked both pieces and chose the best one overall.
We wanted to determine which content would be most helpful for patients — GPT-3, GPT-4 or human-written. Let’s compare the first two pieces (GPT-3 vs. GPT-4).
Phase 1: ChatGPT-3 vs. ChatGPT-4 Healthcare Content
We gave both versions of GPT the same detailed prompts and basic outlines. Both pieces of content followed the outline closely, including the same sections. We chose two sections to compare for this blog post.
Section-to-Section Comparison
Example A: Is a Well-Woman Exam the Same as an Annual Physical?
ChatGPT-3 content:
[h2] Is a Well Woman Exam the Same as an Annual Physical?
[h3] Clarifying the difference
While a well woman exam shares some similarities with an annual physical, there are distinct differences. A well woman exam specifically focuses on women’s health needs, preventive screenings, and addressing concerns unique to women. It provides a comprehensive evaluation of your reproductive and overall health, ensuring that you receive the specialized care you deserve.
ChatGPT-4 content:
[h2] Is a Well Woman Exam the Same as an Annual Physical?
While a well woman exam shares some similarities with an annual physical, they serve different purposes. A well woman exam focuses specifically on women’s health needs and includes screenings, assessments, and discussions tailored to address female-specific concerns. It encompasses reproductive health, breast health, and screenings for cervical and other gynecological conditions. On the other hand, an annual physical is a more general health evaluation that encompasses overall wellness, including blood pressure, cholesterol, and other vital signs.
Example B: How to Prepare for a Women’s Health Exam
ChatGPT-3 content:
[h2] How to Prepare for a Women’s Health Exam
[h3] Getting ready for your visit
To make the most of your Women’s Wellness Exam, it’s important to come prepared. Here are some tips to help you get ready for your visit:
- Gather any relevant medical records or documents.
- Follow any fasting instructions if applicable.
- Write down any questions or concerns you may have.
- Wear comfortable clothing for the exam.
ChatGPT-4 content:
[h2] How to Prepare for a Women’s Health Exam
Preparing for a women’s health exam can help ensure a smooth and productive visit. Here are some essential tips:
- Gather your medical history: Bring relevant medical records, including information about past surgeries, medications, allergies, and family medical history.
- Prepare a list of questions: Jot down any concerns or questions you have for your healthcare provider to make the most of your appointment.
- Follow fasting instructions (if applicable): If your exam requires fasting, ensure you follow the specific instructions provided by your healthcare provider.
- Dress comfortably: Wear loose and comfortable clothing that is easy to remove during examinations.
Being prepared for your women’s health exam empowers you to actively engage in your healthcare journey and make the most of your visit.
Observations and the Winning AI-Written Healthcare Content
The GPT-3 content was vague. It skirted around the differences between an annual physical and well-woman exam, offering a generalized response. The tips for preparing for an appointment were unhelpful and sometimes irrelevant.
ChatGPT-3 content also included redundant subheadings, failed to share custom information about UHealth and missed opportunities for internal links.
GPT-4’s response was more detailed and useful. It clarified the differences between a well-woman exam and an annual physical and gave more specific information about preparing for an exam.
It also included more custom details about UHealth and gave us better link suggestions than GPT-3. While there were still missed opportunities for UHealth-specific content and internal links, the GPT-4 piece was clearly better.
Phase 2: Human-Written Vs. ChatGPT Healthcare Content
It was time to share the winning GPT piece and our writer’s webpage with the SME. See how the two pieces stacked up.
Section-to-Section Comparison
Example A: Is a Well-Woman Exam the Same as an Annual Physical?
ChatGPT-4 content:
[h2] Is a Well Woman Exam the Same as an Annual Physical?
While a well woman exam shares some similarities with an annual physical, they serve different purposes. A well woman exam focuses specifically on women’s health needs and includes screenings, assessments, and discussions tailored to address female-specific concerns. It encompasses reproductive health, breast health, and screenings for cervical and other gynecological conditions. On the other hand, an annual physical is a more general health evaluation that encompasses overall wellness, including blood pressure, cholesterol, and other vital signs.
Human-written content:
[h2] Is a Well-Woman Exam the Same as an Annual Physical?
Well-woman exams and annual physicals are similar, but they aren’t the same. Your annual physical is usually with a primary care provider. This visit includes a checkup of your overall health and counseling on health improvements, just like a well-woman exam. But many primary care providers don’t include pelvic exams, breast exams, or Pap smears in annual physicals.
Example B: How to Prepare for a Women’s Health Exam
ChatGPT-4 content:
[h2] How to Prepare for a Women’s Health Exam
Preparing for a women’s health exam can help ensure a smooth and productive visit. Here are some essential tips:
- Gather your medical history: Bring relevant medical records, including information about past surgeries, medications, allergies, and family medical history.
- Prepare a list of questions: Jot down any concerns or questions you have for your healthcare provider to make the most of your appointment.
- Follow fasting instructions (if applicable): If your exam requires fasting, ensure you follow the specific instructions provided by your healthcare provider.
- Dress comfortably: Wear loose and comfortable clothing that is easy to remove during examinations.
Being prepared for your women’s health exam empowers you to actively engage in your healthcare journey and make the most of your visit.
Human-written content:
[h2] How to Prepare for a Women’s Health Exam
Compiling your family health history is one of the most important ways to prepare for your women’s health exam. Knowing your family history helps your provider give you a more accurate estimate of your risk of certain diseases. You’ll want to find out the following:
- If you have relatives who have had chronic diseases such as heart disease or diabetes
- If you have relatives who have had cancer
- The age when those relatives received the diagnosis
It’s also important to share your personal health history with your provider:
- Any surgeries or major injuries
- Past and current diagnoses
- Medications you take regularly, including over-the-counter medicines and supplements
- Date of your last blood tests and the results
Take some time to consider what questions you have for your provider and what you hope to get out of the visit. Noting your health goals and questions ahead of time sets you up for a more successful appointment.
Your provider’s office may ask you to fill out your family and personal health history online before your appointment. Or you may fill it out at your appointment. On the day of your visit, make sure you bring your photo identification and insurance card.
Final Verdict: Is AI-Written Healthcare Content Helpful?
While GPT-4 included more helpful information than GPT-3, the human-written healthcare content was patient-centered, reader-friendly and driven by expertise.
ChatGPT’s content about the differences between a well-woman exam and an annual physical received a post-graduate readability score on Hemingway Editor. Our writer used shorter, less complex sentences (called plain language), making the information more accessible to everyone.
“Since we had SME input [for the human-written piece], I think we were at an advantage to truly break down for a patient what to expect at women’s wellness exam and the distinction between that vs. your annual physical. In my opinion, this was the best written. I think ChatGPT-4 could be more beneficial for writing canned health library content, but for these ‘What to Expect’ pieces, the [human writer] has the advantage to create better content since they have access to the SME to get the customized information we need.”
Ashley Lewis, Web Content Specialist at UHealth
Our writer capitalized on the SME’s expertise to create content custom to UHealth. She gave rich details from UHealth’s SME about preparing for a women’s exam, including gathering aspects of your family and personal health history to share with your provider.
Here’s the final verdict from an outside perspective — the SME who participated in our experiment:
“I think both versions are written well. V1 [the writer’s piece] is more comprehensive, detailed and took longer to read and review (~12 minutes for me). V2 [GPT-4’s piece] is what I’m thinking was generated by ChatGPT and is accurate. It was factual, straight to the point, less “narrative-like” and quicker to read (~5-6 minutes). I like V1 for its comprehensiveness — in my opinion, it will better prepare the patient.”
Camille Moreno, DO, NCMP
Tips for Helpful Healthcare Content
Stay up to date on information about creating valuable content, both with and without ChatGPT. Receive twice-monthly newsletters about healthcare marketing.
