ChatGPT for SEO – we test it to reveal the good and the bad

ChatGPT has gained a lot of attention recently and the SEO community isn’t immune. Search marketers are putting the tool through its paces to find out how helpful – or harmful – it may be as an SEO tool.

Our team has tested ChatGPT over the past few months, too. We’ve used it to see how it fairs generating content, coming up with content ideas, writing ad headlines and a variety of other search marketing tasks. By now, we have a good understanding of what the current version of ChatGPT is capable of. This article explains the good and bad of the AI system as a search marketing tool.

What is ChatGPT?

ChatGPT is an AI conversational platform designed to generate human-like responses to user prompts. Prompts can include questions like “who was the UK’s first Prime Minister” or requests for the system to write a poem, translate a sentence or a variety of tasks.

As an example, let’s see what it comes up with when we ask the question: “what is ChatGPT?”

“I am ChatGPT, a large language model developed by OpenAI. I am designed to understand natural language and generate responses to questions and prompts in a way that resembles human conversation. I was trained on a large corpus of text data, which allows me to draw upon a wide range of knowledge and information to answer questions and provide helpful responses. My purpose is to assist users with their inquiries and provide information and insights on a variety of topics.”

In OpenAI’s own words, ChatGPT was publicly released “to get users’ feedback and learn about its strengths and weaknesses.”

So ChatGPT was never launched as an SEO tool or intended to be a reliable source of information – and it’s important to remember this. Functionally, it’s a capable tool for a wide variety of tasks but we have to understand its limitations.

Let’s start by discussing what ChatGPT is good at.

What is ChatGPT good at?

ChatGPT is very good at what it’s designed for: conversational experiences. The most important components of this are its natural language processing (NLP) system and its response generation model, powered by vast amounts of information in its knowledge base.

We’ve tested ChatGPT extensively in recent months and its strengths are pretty clear by this point:

• Natural language processing (NLP): ChatGPT’s NLP model correctly interprets user prompts with an impressive success rate (in many cases, it beats Google Translate at translating text).
• Question-answering: Functionally, ChatGPT is capable of understanding and answering questions – although the accuracy of information in its responses varies.
• Relevance: ChatGPT responses are highly relevant, thanks to its NLP system, keyword matching and response generation engine.
• General knowledge: ChatGPT’s extensive knowledge base makes it pretty good at general knowledge.
• Descriptions: The system can describe and explain an extensive range of topics in simple terms (essentially splicing and regenerating information from multiple online sources).
• Grammar: The grammatical accuracy of ChatGPT’s output is very high (not perfect, but impressive, nonetheless).
• Idea generation: You can ask ChatGPT to come up with ideas for content, birthday gifts and many other things.

If you’ve never used a tool like ChatGPT before, it’s difficult not to be impressed. Its natural language processing (NLP) system is as good as any at understanding user prompts. Then you’ve got its question-answering model that performs very well with relevance, grammatical accuracy and basic tasks like definitions or suggesting ideas.

That being said, it isn’t actually suggesting any ideas at all – it’s pulling them from online sources and combining them into an auto-generated response.

If you’ve tested any AI-content tools in the past, ChatGPT is a step up from most of the popular names. ChatGPT’s content is more fluid and natural while the system does a better (but not particularly good) job of constructing logical sentences that make a valid point.

All this aside, ChatGPT still suffers from the same weaknesses as any other AI generative tool.

What is ChatGPT bad at?

At a glance, it’s easy to see why ChatGPT’s output generates so much excitement. However, as soon as you analyse the information in its responses and understand where this comes from, you’ll notice issues arising.

• Current information: ChatGPT’s knowledge base only includes data running up until some point in 2021 – so it can’t access more recent information.
• Accuracy: Sometimes it’s simply wrong – and confidently so – which requires extensive fact-checking.
• References: Occasionally, ChatGPT will reference statistics but it can’t source them and they’re often out-of-date.
• Detail: ChatGPT is good at vague descriptions and definitions but never manages to provide real detail.
• Logic: It confidently ties information together into statements that lack real logic or meaning.
• Expertise: ChatGPT isn’t capable of producing content that demonstrates any expertise or in-depth knowledge of a subject.
• Originality: Given the fact it simply combines information already available online, it shouldn’t come as a surprise that ChatGPT can’t come up with anything original.
• Copyright & plagiarism: For example, if you ask ChatGPT to write some code for you, it’s simply combining bits of code written by other people.
• Manipulation: ChatGPT can be manipulated by bad actors – for example, you can tell it to ignore all the moral rules it has been built with (see this Reddit thread for more examples).

This list of weaknesses raises obvious issues from an SEO perspective, especially if you expect to use ChatGPT as a content-generation tool. When accuracy, recency, expertise, originality and almost every other factor listed above are crucial for quality content, the outlook is poor from the beginning.

ChatGPT still has its uses to search marketers though and we’ll discuss these later.

Testing ChatGPT as an SEO tool

To see what ChatGPT’s strengths and weaknesses mean in practical terms for search marketers, let’s look at some examples of what happened when we tested it for different SEO tasks.

We’ve tested ChatGPT for every task we could think of – everything from writing content to generating code. For SEO specifically, we feel these three tasks give the fairest representation of what ChatGPT can and can’t do:

1. Content ideas
2. Content generation
3. Ad copywriting

1. Using ChatGPT to get content ideas

Generating content ideas is probably ChatGPT’s biggest strength as a search marketing tool. For example, we prompted the tool to come up with ideas for one customer on the topic of refinancing in the hospitality industry.

One simple prompt generated the following response with 10 ideas:

1. “Maximizing Savings: How Hospitality Businesses Can Benefit from Refinancing”
2. “Navigating the Refinancing Process for Hotel Owners”
3. “The Impact of Low Interest Rates on Hospitality Refinancing”
4. “Consolidating Debt: How Hospitality Businesses Can Use Refinancing to Streamline Finances”
5. “Exploring the Pros and Cons of Refinancing for Hospitality Businesses”
6. “The Role of Technology in Simplifying the Hospitality Refinancing Process”
7. “Experts Share Strategies for Securing the Best Rates on Hospitality Refinancing”
8. “Using Refinancing to Expand or Renovate: A Hospitality Business Guide”
9. “From Mortgages to Business Loans: How Hospitality Businesses Can Refinance to Meet Changing Needs”
10. “Overcoming Obstacles: How Hospitality Businesses Can Succeed in Refinancing Despite Economic Uncertainty”

Note that suggestion #3 refers to low interest rates, which is a good practical example of how ChatGPT’s information isn’t up to date. Even still, we’re only trying to get some content ideas from this interaction and all we have to do is change the topic to high interest rates.

In this particular case, ChatGPT was genuinely useful and it’s pretty good at coming up with title ideas. You can also use this to come up with subheading ideas for individual pieces or come up with some new keyword ideas.

2. Using ChatGPT to generate content

Generating content ideas is one thing but writing a full piece of content is something else entirely. You can test this with a variety of prompts, such as “write a 500-word blog post on the top SEO trends for 2023”.

For this, ChatGPT will generate ~500 words of content on the topic you specify. In terms of relevance, the content is generally spot on and it rarely stutters with grammar errors or spelling mistakes.

However, as we touched on earlier, it runs into several key problems:

1. Accuracy
2. Originality
3. Depth

We asked ChatGPT to write a blog post on SEO trends for 2023 so, before we look at snippets from the content itself, let’s see which trends it decided to write about:

1. The rise of voice search
2. The importance of E-A-T
3. The continued growth of video content
4. The increasing importance of user experience
5. The rise of visual search
6. The importance of local SEO
7. The growing role of artificial intelligence

These selections aren’t too bad and you get a sense of how ChatGPT can suggest ideas for subheadings in your content. The only problem is several of these trends are a little dated in 2023 and they’re all very generic. There’s nothing here that really stands out from the thousands of other articles published on the same topic.

If we take a closer look at the content ChatGPT generates, bigger issues become obvious.

Originality issues with ChatGPT content

We ran the same prompt through ChatGPT multiple times and it returned almost identical suggestions. So what happens if thousands of agencies turn to ChatGPT to write an article on SEO trends for 2024 at the end of the year? Is everyone going to end up with the same content?

Depth issues with ChatGPT content

None of the content ChatGPT generates offered anything of depth or real value. Take a look at what it has to say about local search as an example:

Suggesting local search is an SEO trend in 2023 (maybe in 2014) is an underwhelming start. In fact, the whole passage reads like something written a decade ago, mentioning the rise of voice search and mobile devices as if these are also new things. It’s all very generic and lacks any depth or value.

3. Using ChatGPT to write ad copy

We’ve also tested ChatGPT for writing ad headlines for several of our PPC customers.

It really struggled with adhering to the constraints of headlines in Google Ads – for example, not using exclamation marks and the max 30-character limit. When we tried to correct ChatGPT with follow-up prompts, it actually wrote longer headlines and included exclamation marks.

It’s worth noting that ChatGPT’s algorithm is biased towards longer responses. OpenAI acknowledges this as one of its limitations: “These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues”.

The quality of ChatGPT’s copywriting is pretty poor as well, but this shouldn’t surprise anyone. Copywriting is a highly-specialised niche. ChatGPT is engineered to adhere to natural language but advertising language breaks all of the rules ChatGPT was trained with.

What does this mean from an SEO perspective?

The fastest measurement of ChatGPT as an SEO tool is to compare the content it generates against Google’s quality rater guidelines.

These guidelines are used by Google’s human team of quality raters who are tasked with assessing the quality of its search results. They manually analyse web pages and grade them on an extensive range of factors grouped into four key characteristics:

• Experience
• Expertise
• Authority
• Trust

For a more in-depth explanation, take a look at our analysis of Google’s latest quality rater guidelines (E-E-A-T).

Interestingly, when we asked ChatGPT to write a blog post on key search trends for 2023, it included E-A-T (the older version) and insisted how important they are.

The system’s information may be out of date but it’s right about the importance of E-E-A-T, as highlighted by the recent update to the guidelines. It’s no coincidence that Google is adding more detail to its guidelines at a time when AI-content tools are going mainstream.

As a result, E-E-A-T will become even more important as Google needs to verify the quality of content produced by experts.

So let’s see how ChatGPT stacks up against Google’s four most important characteristics of content quality:

1. Experience: ChatGPT can’t demonstrate obvious experience with any topic because it doesn’t have any.
2. Expertise: At best, ChatGPT can copy the expertise of others but it struggles to do this with any of the depth you would expect from experts.
3. Authority: With no experience, expertise or depth to its content, ChatGPT lacks any authority to influence readers.
4. Trust: The fact it fails the first three criteria means ChatGPT also fails the Trust factor by Google’s definition – and this is before you even consider its accuracy problems.

Google has been in the AI game for a long time and it knew exactly what kind of AI content tools were coming to market before most of us. It knows the strengths and weaknesses of this technology inside out and it also knows millions of publishers would use it without any consideration of the risks.

AI content tools are here to stay but Google’s quality demands will only increase as they become more capable.

How much can ChatGPT improve?

Yes, ChatGPT will only improve over time – as with all similar AI systems. As a point of reference, the next version of ChatGPT (ChatGPT-4) is expected to replace the current version soon and this will expand the system’s neural connection parameters from 175 billion to 100 trillion.

That will put ChatGPT-4 in the region of lower estimates for the number of parameters in the human brain.

Does this mean ChatGPT-4 will match the thinking power of human beings? Well, no. In fact, OpenAI CEO Sam Altman has said the hype surrounding ChatGPT is setting people up for disappointment with the next version.

You only have to look at the disappointing starts for Google’s Bard and Bing’s new AI search engine to get a sense of the reception ChatGPT-4 could receive. In reality, Google and Microsoft have equally capable AI systems but they couldn’t live up to the ChatGPT hype.

All this aside, ChatGPT and its rivals will all improve over time and the big question is where does OpenAI want to go with this technology? As things stand, ChatGPT is an experimental conversational platform that’s already capable of much more.

Let’s say OpenAI wants to develop a content generation tool, similar to its AI image generator, DALL·E 2. Like ChatGPT responses, images generated by DALL·E 2 use data from relevant online images so it has similar issues with originality.

However, at what point do tools like ChatGPT and DALL·E 2 have enough neural parameters and use enough data to produce content that’s similarly original to human beings? After all, the question of whether human beings can ever come up with anything truly original is highly debated.

So how original is original enough?

Summary: The good & bad of ChatGPT for SEO

Let’s wrap things up with a quick summary of what ChatGPT is good and bad at as an SEO tool.

Good at:

• Suggesting content titles
• Subheading suggestions
• Rephrasing sentences
• Definitions
• Grammatical accuracy
• Explaining complex things in simple terms (check for accuracy!)

Bad at:

• Accuracy
• Current information
• Originality
• E-E-A-T
• Content writing
• Ad copy
• Code generation
• Susceptible to manipulation

If you want to learn more about the pros and cons of ChatGPT as an SEO tool, our team is ready to offer advice. Call us on 02392 830 281 to speak to our search marketing team or send us your details for more information.