What are Google's Search Quality Rater Guidelines?
A bit of context and history.
Google's Search Quality Evaluator Guidelines is the document with instructions for Google raters on how to rate the quality of web pages and the relevance of search results. The key points of that document are provided here.
It's possible to find online references to Google's search quality rating program and guidelines that dates back to 2005, and even 2003, which proves that the company has been using this type of human feedback on search quality for over 15 years! But for a long period of time, the raters' guidelines were kept strictly confidential.
Despite Google's attempt to keep those guidelines secret, so to say, they would always end up being leaked.
In 2013, for the first time, Google decided to share their Search Quality Rating Guidelines publicly to the world. A big step, but with caveats.
The public version of the document was only 43 pages, whereas the leaked, non-public version was 161 pages. They had gutted the guidelines for public release!
It was only in 2015 that the full version of the document was finally made publicly available by the company.
Ever since, in numerous opportunities and especially after the release of major updates, Google is actively recommending web publishers to read the quality rater guidelines.
"If you understand how raters learn to assess good content, that might help you improve your own content. In turn, you might perhaps do better in Search.", states Google on its recent guidance post on core algorithm updates.
How are the Guidelines Structured?
The document is divided into three parts.
Part 1: Page Quality Rating Guideline
This is where Google specifies the criteria associated with page quality rating. Raters are instructed to observe several factors that are important as quality indicators, with special attention to Expertise, Authoritativeness and Trustworthiness (E-A-T). Examples are provided to illustrate each rating.
Part 2: Understanding Mobile User Needs
To determine the relevance rating of a search result, the first step is understanding the needs of the user. This is where Google provides guidelines on what is important to consider for understanding user needs, particularly with mobile users in mind.
Part 3: Needs Met Rating Guideline
After understanding the needs of the user, it's possible to assess how helpful a result is to satisfy their intent (i.e. meet the user needs). This is where Google provides additional guidelines on what is important for relevance rating. Part 3 is highly based on rating examples.
What changes, and how often?
There is not a predefined update frequency to the quality rater guidelines. Apparently, Google updates the document on an ad hoc basis.
According to historical data, we can expect up to two updates per year on average.
The changes are usually small and subtle. For example, they might include or remove examples, rephrase some sentences here and there, bring more clarity to what could be considered vague or confusing, provide additional instructions to avoid any bias, etc.
In other words, there is very little, if any, changes to core of the document's content -- that is, on the guidelines as to how to evaluate web page quality and search result relevance. What changes is content that can be considered supplementary to the core content.
Either way, these updates always resonate heavily within the SEO community. And there is a reason for that: changes in the guidelines may signal that Google could be adjusting their algorithms to target those amendments.
What changed in its latest update from Sep. 05, 2019
No matter if minor or major, the SEO community will make the updates insightful.
"The rating guidelines have been remarkably accurate for predicting the algorithm trends", says SEO consultant Roger Montti on his piece for Search Engine Journal about the latest guidelines update. Montti points out that in past updates, "the increased instructions on how to rate medical and financial sites, for example, coincided with algorithms designed to improve the relevance of those kinds of sites."
Simply put, when the guidelines change to provide instructions on how to approach certain specific rating situations, it likely means that Google is testing algorithm changes that address those specific cases before releasing the updates in live search results.
In its latest update, mostly minor changes were observed in a couple of sections. The main changes and takeaways are summarized below:
- On Section 2.3, Google updated the list of examples for YMYL topics. The topic News and Current Events now has its own dedicated section placed at the top of the list of examples. They included the word "topic" or "topics" in some sentences where they were previously using just the word "page" or "pages". The "Other" section on the list of YMYL examples has been expanded to include additional topics.
- Takeaways: According to SEO specialist Lily Ray, "placing News and Current events above all YMYL topics may indicate that Google is attempting to increase the priority and importance of news as a YMYL category." Ray also says that "using the word 'topic' over 'page' may hint at the fact that some YMYL content on a page can be enough for the page to considered YMYL, even if that is not the overall purpose of the page." Roger Montti finds notable that the “Other” topic has been expanded and speculates: "Will topics like Fitness & Nutrition, College Search and Job Search receive scrutiny and emphasis in a future round of core algorithm updates? Those are big money affiliate topics. It may also be reasonable to assume that the quality raters hadn’t been focusing enough on these topics," he adds.
- On Section 2.6.1, Google now added that "a history of high quality original reporting are strong evidence of a positive reputation", in a paragraph where they previously only referred to "prestigious awards", such as the Pulitzer Prize, as strong evidence of positive reputation
- Takeaways: Lily Ray believes that Google is supplementing any reference of the Pulitzer Prize with other examples that could indicate journalistic quality because the company faced backlash from conservative media, which claims the Pulitzer Prize favors liberal publications. "This change may have been an attempt for Google to appear as more politically neutral," states Ms. Ray. For SEM consultant Jennifer Slegg, these changes might be due to the fact that "perhaps raters were taking this far too literally about a news site requiring a Pulitzer or other award to have a high reputation."
- On Section 5, a broad range of content was added to illustrate the concepts behind Highest Quality pages and examples of such pages. Emphasis was given to the importance of uniqueness and originality as factors that distinguish very high quality main content, and examples were included on what constitutes original content and reputation signs depending on the type of website. The added content also stresses the importance of applying high quality standards for other types of content beyond text, like images and videos, for example.
- Takeaways: Apparently, Google is aiming at bringing more specificity and clarity on what defines highest quality pages depending on the type of website. This specificity helps not only evaluators to assign the best possible quality rating to a page, but also content creators who can now follow more specific guidelines on what constitutes highest quality characteristics for their niche.
What no one tells you about the Quality Rater Guidelines
The level of detail that many SEO's and marketers put into analyzing the Google rater guidelines is truly impressive. Few documents in the history of SEO and content marketing have been scrutinized to such extent.
But there's a big but here. All the most prominent consultants who are currently regarded as references in this topic have never worked as raters!
How would it be asking for legal advice from a person who is very knowledgeable of the guidelines for consumer protection but has never worked as attorney? That would be legitimate, of course, but only a professional would be able to know how that document works in the context of a legal process.
This exact analogy applies to what is going on, over the last years, in the SEO industry regarding the search quality rating guidelines. Admirable professionals who dive in deeply into every detail of the guidelines, but who are completely unfamiliar with the search quality rating process.
So, let's clarify a few things.
Google's Quality Evaluator Guidelines are general guidelines, which means that in many cases the document is broad, vague and ambiguous, especially in the Needs Met section.
For example: how to evaluate the influence that the position of the target content on the page has on its relevance rating for the query? In our SEA Model methodology, we call this Extraneousness Issue.
And what about those extra clicks that may be requested from the user, or too much elapsed time before they are able to find the target content? What kind of influence do such things have in the relevance rating? In the SEA Model, that's what we call Effort Issue.
Experienced, top performing raters also know that a result which addresses a less likely intent of the query cannot be rated in the higher end of the relevance scale, no matter the amount of content it provides for that intent. This is not something you will ever see addressed by SEOs who know the guidelines but don't have enough or any experience as a rater.
You see, even with their 167 pages, the guidelines are far from specific enough to address many of these fairly common rating situations. Hence the term General Guidelines.
Google Quality Raters Guidelines for SEO and Content Marketing
SEO consultants and marketers who neglect the raters' guidelines or claim that the document is overrated probably didn't realize that modern SEO is Content SEO, as most platforms are increasingly automated and technically optimized for search engines.
On that premise, the Search Quality Evaluator Guidelines are a treasure for SEO and Content strategists, especially if used in conjunction with techniques known only to experienced, professional search quality raters.
The main reasons that make the document a gold mine of SEO and Content information are as follows:
- It comes directly from Google
- It provides details on what high quality pages should look like
- It explains how the relevance of search results should be assessed
- It shows where Google wants the search algorithm to go, according to Ben Gomes, the man in charge of Google Search
- It shows what Google considers great content, which is a fundamental requirement for doing well in Search, as pointed out by Google Search Liaison Danny Sullivan
Where to get Raters' Guidelines training and consultancy?
At the Search Evaluator Academy, there is a course based on quizzes created from the content of the guidelines. This can be useful for SEOs and marketers looking to ensure their understanding of that document.
There is also the SEA Model course, which is a complementary methodology for search evaluation to be used in conjunction with the general guidelines. The SEA Model is designed to make you to think like an experienced rater, even if you have absolutely no experience in this field.
The Search Evaluator Partners program is a consulting service tailored to digital marketing agencies. It provides an opportunity for them to learn in depth how search evaluation can help with SEO and Content, all while we evaluate the websites of their own clients or projects.