Before we begin, let’s make one thing clear — we’ve always stood behind human-first, human-written content at Aha Media Group. We know our biases as a premium content agency. And that’s exactly why we decided to test them.
In the wake of Google’s lightened stance on AI-generated healthcare content (read the latest guidelines on helpful content) we ran an experiment.
We wanted to see how ChatGPT-3 and ChatGPT-4 healthcare content compared to human-written content. So we put them to the test. First, ChatGPT-3 vs. 4, then the winning piece vs. a human writer’s content.
Here’s how we set it up.
To keep things fair, we gave ChatGPT-3, ChatGPT-4 and our writer the same assignment: Write a webpage for the University of Utah Health Women’s Health Services. We looped in UHealth for feedback on all 3 pieces.
The human-written content went through our usual process. Our writer interviewed a subject matter expert (SME) at UHealth and worked with our editor to send a polished piece.
For the AI-generated content, we used prompts with specific instructions for both GPT-3 and GPT-4. Here was our first prompt:
A series of follow-up prompts shared the information we include in content briefs for writers:
We also gave GPT-3 and GPT-4 detailed instructions that we’d expect writers to know:
See the outline:
Our last ChatGPT healthcare prompt asked GPT-3 and GPT-4 to suggest internal links within the page to other UHealth pages.
Aside from proofreading, we sent the AI healthcare content to UHealth without human intervention. UHealth then reviewed and edited. The GPT-3 piece took 56 minutes to edit and GPT-4 took 32 minutes. UHealth shared feedback on both pieces and chose the best one to compare with our writer’s.
Then, UHealth ran the winning ChatGPT healthcare content and the human writer’s piece through their usual review process with a doctor who didn’t know which was which. The doctor fact-checked both pieces and chose the best one overall.
We wanted to determine which content would be most helpful for patients — GPT-3, GPT-4 or human-written. Let’s compare the first two pieces (GPT-3 vs. GPT-4).
We gave both versions of GPT the same detailed prompts and basic outlines. Both pieces of content followed the outline closely, including the same sections. We chose two sections to compare for this blog post.
[h2] Is a Well Woman Exam the Same as an Annual Physical? [h3] Clarifying the difference
While a well woman exam shares some similarities with an annual physical, there are distinct differences. A well woman exam specifically focuses on women’s health needs, preventive screenings, and addressing concerns unique to women. It provides a comprehensive evaluation of your reproductive and overall health, ensuring that you receive the specialized care you deserve.
[h2] Is a Well Woman Exam the Same as an Annual Physical?
While a well woman exam shares some similarities with an annual physical, they serve different purposes. A well woman exam focuses specifically on women’s health needs and includes screenings, assessments, and discussions tailored to address female-specific concerns. It encompasses reproductive health, breast health, and screenings for cervical and other gynecological conditions. On the other hand, an annual physical is a more general health evaluation that encompasses overall wellness, including blood pressure, cholesterol, and other vital signs.
[h2] How to Prepare for a Women’s Health Exam [h3] Getting ready for your visit
To make the most of your Women’s Wellness Exam, it’s important to come prepared. Here are some tips to help you get ready for your visit:
[h2] How to Prepare for a Women’s Health Exam
Preparing for a women’s health exam can help ensure a smooth and productive visit. Here are some essential tips:
Being prepared for your women’s health exam empowers you to actively engage in your healthcare journey and make the most of your visit.
The GPT-3 content was vague. It skirted around the differences between an annual physical and well-woman exam, offering a generalized response. The tips for preparing for an appointment were unhelpful and sometimes irrelevant.
ChatGPT-3 content also included redundant subheadings, failed to share custom information about UHealth and missed opportunities for internal links.
GPT-4’s response was more detailed and useful. It clarified the differences between a well-woman exam and an annual physical and gave more specific information about preparing for an exam.
It also included more custom details about UHealth and gave us better link suggestions than GPT-3. While there were still missed opportunities for UHealth-specific content and internal links, the GPT-4 piece was clearly better.
It was time to share the winning GPT piece and our writer’s webpage with the SME. See how the two pieces stacked up.
[h2] Is a Well-Woman Exam the Same as an Annual Physical?
Well-woman exams and annual physicals are similar, but they aren’t the same. Your annual physical is usually with a primary care provider. This visit includes a checkup of your overall health and counseling on health improvements, just like a well-woman exam. But many primary care providers don’t include pelvic exams, breast exams, or Pap smears in annual physicals.
Compiling your family health history is one of the most important ways to prepare for your women’s health exam. Knowing your family history helps your provider give you a more accurate estimate of your risk of certain diseases. You’ll want to find out the following:
It’s also important to share your personal health history with your provider:
Take some time to consider what questions you have for your provider and what you hope to get out of the visit. Noting your health goals and questions ahead of time sets you up for a more successful appointment.
Your provider’s office may ask you to fill out your family and personal health history online before your appointment. Or you may fill it out at your appointment. On the day of your visit, make sure you bring your photo identification and insurance card.
While GPT-4 included more helpful information than GPT-3, the human-written healthcare content was patient-centered, reader-friendly and driven by expertise.
ChatGPT’s content about the differences between a well-woman exam and an annual physical received a post-graduate readability score on Hemingway Editor. Our writer used shorter, less complex sentences (called plain language), making the information more accessible to everyone.
“Since we had SME input [for the human-written piece], I think we were at an advantage to truly break down for a patient what to expect at women’s wellness exam and the distinction between that vs. your annual physical. In my opinion, this was the best written. I think ChatGPT-4 could be more beneficial for writing canned health library content, but for these ‘What to Expect’ pieces, the [human writer] has the advantage to create better content since they have access to the SME to get the customized information we need.” Ashley Lewis, Web Content Specialist at UHealth
“Since we had SME input [for the human-written piece], I think we were at an advantage to truly break down for a patient what to expect at women’s wellness exam and the distinction between that vs. your annual physical. In my opinion, this was the best written. I think ChatGPT-4 could be more beneficial for writing canned health library content, but for these ‘What to Expect’ pieces, the [human writer] has the advantage to create better content since they have access to the SME to get the customized information we need.”
Ashley Lewis, Web Content Specialist at UHealth
Our writer capitalized on the SME’s expertise to create content custom to UHealth. She gave rich details from UHealth’s SME about preparing for a women’s exam, including gathering aspects of your family and personal health history to share with your provider.
Here’s the final verdict from an outside perspective — the SME who participated in our experiment:
“I think both versions are written well. V1 [the writer’s piece] is more comprehensive, detailed and took longer to read and review (~12 minutes for me). V2 [GPT-4’s piece] is what I’m thinking was generated by ChatGPT and is accurate. It was factual, straight to the point, less “narrative-like” and quicker to read (~5-6 minutes). I like V1 for its comprehensiveness — in my opinion, it will better prepare the patient.” Camille Moreno, DO, NCMP
“I think both versions are written well. V1 [the writer’s piece] is more comprehensive, detailed and took longer to read and review (~12 minutes for me). V2 [GPT-4’s piece] is what I’m thinking was generated by ChatGPT and is accurate. It was factual, straight to the point, less “narrative-like” and quicker to read (~5-6 minutes). I like V1 for its comprehensiveness — in my opinion, it will better prepare the patient.”
Camille Moreno, DO, NCMP
Stay up to date on information about creating valuable content, both with and without ChatGPT. Receive twice-monthly newsletters about healthcare marketing.
Sign Up
