The following guideline applies to the next set of tasks you will see.TAKE A BREAK
Search Ads Evaluation: Landing Page Only
READ INSTRUCTIONS CAREFULLY!
These ratings use newly updated ads rating guidelines that are different from earlier versions of the guidelines. If you have not read the guidelines since the version date listed above, you must carefully read or review all instructions before rating.
Search ad rating involves interpreting a user query. A user query is the set of keywords that a user enters into the Google search engine. When rating a search ad’s landing page, perform the following steps:
- Review the Google search results page, try to understand the user query, and form an opinion about what the user hopes to accomplish by using a search engine.
- Use the evaluation criteria found in the following instructions to analyze the advertising experience the user will have on the landing page.
An understanding of the user intent is necessary to accurately rate a search ad’s landing page. The user intent is what the user hopes to accomplish by using the Google search engine. Note that users use the search engine to look for a variety of things, and there are many user intents.
Some queries are very easy to understand, others are more difficult, and some may seem impossible to understand. Regardless of its meaning, you must research the query and form an opinion about the user intent. We strongly advise you to review the Google search results page to determine user intent. In order to objectively determine how promising or unpromising an advertiser offering is for a particular user query, it is important to form an opinion about the user intent before beginning an analysis of the advertisement.
Queries with Multiple Meanings
If a query has multiple meanings, please consider that all meanings can be placed on a spectrum between plausible meanings and highly implausible meanings. When analyzing an advertiser offering, consider what meaning the advertiser uses and where it falls along this spectrum. This will help you determine the appropriate landing page rating.
If a query has several plausible meanings, it is important to consider them all. If an advertiser assumes a particular meaning on a landing page, and it is a reasonable meaning, assume that is the meaning that the user intended.
Refer to the following example to better understand plausible meanings:
User query: [ java ]
This query could refer to an island, coffee, or a computer language. With no additional information available, it is impossible to say which meaning the user intended. Landing pages that respond to any of these meanings are acceptable since all three meanings are reasonably plausible.
Possible but Unlikely Meanings
If a landing page assumes a meaning that is possible but not very likely, this a secondary interpretation. A landing page that addresses only a secondary interpretation of the query is given a lower rating than a landing page that addresses a plausible meaning. Use the Secondary Interpretation of Query flag in this case. Generally, rate a landing page that responds to a secondary interpretation negatively.
Refer to the following example to better understand possible but unlikely meanings:
User query: [ paris ]
While there are a number of cities called Paris, unless there is some reason to think that one of the smaller cities is meant, a query mentioning Paris is probably referring to Paris, France. So, a landing page for hotels in Paris, Texas, instead of Paris, France, is probably incorrectly comprehending the user intent. Even if the landing page is otherwise a good one, rate it on the negative side of the scale and use the Secondary Interpretation of Query flag.
If a landing page assumes a meaning of the query that is completely implausible, treat it as completely wrong and choose a very negative rating. Do not use the Secondary Interpretation of Query flag if the meaning is clearly implausible.
Refer to the following example to better understand implausible meanings:
User query: [ paris ]
The query probably refers to the city of Paris, France. If an advertiser interprets the meaning to be plaster of paris, it is almost certainly not addressing the user query. Use a very negative rating.
Users often misspell queries. When evaluating a query, if it is clear what the user means, and the misspelled version of the query has no meaning, ignore the misspelling.
Analysis is more difficult if the query appears to be a misspelling, but the misspelled version has a unique meaning. First consider the query as the user entered it, and then consider if it may be misspelled. If advertisers respond to misspellings, ratings may need to be adjusted.
Refer to the following example to better understand misspelled queries:
User query: [ goodnight moom ]
There is a famous children’s book called Goodnight Moon. It is very possible that the user means to type [ goodnight moon ] but types [ goodnight moom ] instead. However, there is actually a novel titled Goodnight Moom. While the novel is quite obscure, it might be what the user wants.
Advertiser Responds to Actual Spelling in Query
If the advertiser assumes that the query is correct as it stands (in the example above, assumes the user meant [ goodnight moom ]), treat the advertiser’s query interpretation as acceptable. You would then need to decide separately how promising the landing page is.
Advertiser Responds to a Plausible Correction of Spelling in Query
If the advertiser assumes that the query is misspelled and addresses a corrected version of the query (in the example above, assumes that the user mistyped and meant [goodnight moon]), judge for yourself whether this was a good assumption. If you think it was a reasonable assumption, the landing page is treated as if this were the user intent. Don’t modify your scores to account for the spelling correction, and don’t use the Secondary Interpretation of Query flag.
Advertiser Responds to a Possible but Unlikely Correction of Spelling in Query
If the advertiser assumes a corrected spelling that you think is possible but not very likely, this is a secondary interpretation. A landing page that addresses only a secondary interpretation of the query is given a lower rating than if it had responded to a likely or plausible meaning. The Secondary Interpretation of Query flag must be used. A landing page that responds to a secondary interpretation is generally rated negatively.
Implausible Spelling Correction
If the advertiser’s interpretation of the query is based on a completely unreasonable assumption, treat it as completely wrong and give it a very negative score. The Secondary Interpretation of Query flag is not used in this case.
Continuing with the previous [ goodnight moom ] example, both query interpretations are reasonable. The only way to know this is to research the query and analyze how the advertiser interprets it.
Queries for which a Reasonable Landing Page is Impossible
Sometimes a query is either so hard to interpret or so non–commercial in nature that no landing page will be a good match. Be careful in these cases—rate the landing page according to how well it actually responds to the query, and do not worry about how hard it would be to show an appropriate landing page for that query. Do not give a landing page positive ratings if a better landing page for the query cannot be determined or if it seems like it was a good try. Rate it positively only if it addresses the query intent. If the query intent cannot be determined, the landing page must be rated negatively. For example, a query of [ www ] or [ when did ] is not complete enough to serve a proper landing page.
In some rare cases, a query may appear that is the result of an error in how the task was added to the evaluation system. For example, a query may appear in the incorrect rating language, or a query of jumbled characters may appear that, after research, has no discernible meaning. Do not attempt to provide ratings for queries like this. Instead, use one of the flags provided in the Query Flag section.
When Location Matters
Sometimes the query, the ad, or both may refer to a specific geographic location. Information about the location of the user who entered the query is not available, so it can be difficult to determine how to handle a location–specific advertisement.
Query Specifies a Location but Ad Does Not
If a query specifies a location, take it into account when evaluating the ad. Sometimes research is required to determine whether the product or service is compatible with the location. This research is required before you can choose an Ad Creative rating score.
User query: [ pizza Santa Monica ]
If an ad pointed to the main Round Table Pizza chain homepage, but didn’t mention this California city, this might initially seem useful. However, if, upon using the location finder on the site, there are no locations within a reasonable distance of Santa Monica, this ad is probably not useful.
Ad Specifies Location but Query Does Not
Evaluate the ad as if the user were in an appropriate location. For example, if the query is [ pizza ] and the ad is for a pizza restaurant in Barstow, California, assume that the user was in Barstow, California.
Neither Query Nor Ad Specify a Location
Assume that the user can be anywhere in the target country of the rating language.
Location Mismatch Between Query and Ad
Assuming there is a match between the product or service and the query and ad, take the location proximity into account in evaluating an ad. If the ad for a given query specifies a different and incompatible location, this makes it a worse ad.
User query: [ pizza Santa Monica ]
If the ad is for a pizza restaurant in Manhattan, this is very unpromising. However, an ad returned for pizza in a different but nearby location, like a neighboring town, could useful, and this ad might not be as bad as the previous example. Use common sense to determine if the ad exceeds a reasonable distance for the user, and an acceptable distance may vary depending on the query. For example, a user may be willing to travel farther to buy a new car than to get a haircut or go to the supermarket. For certain queries, serving an ad with a completely different location may still be promising. For example, if a user in the United States is looking for [ vacation in Australia ], then an ad for “vacation in New Zealand” is not necessarily a bad ad since the user is likely to be willing to travel a long distance for a vacation.
Search Ad Rating: Does the Landing Page Satisfy User Intent?
Use the slider bar to select from the following four rating categories while determining how likely it is that a landing page will satisfy the user intent. Only consider the user query and the landing page.
- Satisfaction Likely
- Satisfaction Possible
- Dissatisfaction Possible
- Dissatisfaction Likely
Satisfaction Likely and Satisfaction Possible are positive ratings that satisfy the user query. Dissatisfaction Possible and Dissatisfaction Likely are negative ratings that do not satisfy the user query.
Only rate the landing page that opens after clicking on the the Visit Landing Page button. NEVER copy and paste a link to visit the page, and NEVER manually change the URL.
The fundamental principle of landing page evaluation is this: the user starts a search on Google.com with a goal in mind. The user then enters a query and reviews Google’s search results and ads. The user then clicks on an ad, and that ad takes the user to the landing page. When analyzing a landing page, remember that the user should be closer to his or her goal after clicking the ad. Carefully review the Google search results page to determine this.
Distance from the User’s Goal
Does the Landing Page move user closer to his or her goal, further from the goal, or neither closer nor further from the goal?
If the user is closer to the goal, the landing page deserves a positive rating. For example, if the user is hoping to buy a specific camera, and the landing page is a store offering that camera for sale, the user has come closer to accomplishing his or her goal.
If the user is further from the goal, the landing page deserves a negative rating. If the user is hoping to buy a specific camera, and the landing page is a store offering pet food, this is a dead end. The user will need to go back to the search page or start a new search, so he or she is actually further from the goal than before clicking on the ad.
If the user is neither closer to nor further from the goal, the landing page deserves a negative rating. If the user is on a Google search results page and clicks on an ad that just takes them to a page of similar search results, which overall did not provide any additional value, no progress has been made; the user is no closer to or further from the goal than before clicking the ad.
Deciding this is not an exact science. Rely on good judgment. The following guidelines more deeply explain how to generally rate landing pages, but they do not explain how to rate a landing page in every situation.
Distinguishing Between Satisfaction Likely and Satisfaction Possible
Satisfaction Possible and Satisfaction Likely are positive ratings. If there is a good chance that the search ad will offer the user exactly or very nearly what he or she wanted, use a Satisfaction Likely or Satisfaction Possible rating.
To receive this rating, a landing page must offer just what the user looked for. If the user wants car reviews, it should offer car reviews. If the user wants car reviews about a specific model, it should offer car reviews about exactly that model. If the user wants a category of product, the landing page should be devoted to or include that exact category of product. For a Satisfaction Likely rating, what the user is looking for should be apparent with no additional action needed by the user. It is permissible, however, to click on a link to get detailed information.
Use this rating if the page is satisfactory but does not immediately present exactly what the user seeks. If the product or service is for sale on the site, but a search or straightforward navigation is required to find the item, select a rating of Satisfaction Possible rather than Satisfaction Likely. If the site offers a very plausible substitute for a particular product specified in the query, it may receive a rating of Satisfaction Possible or lower. If the query is a search for information, and this information can be found without too much trouble on the advertiser site but is not on the landing page, use Satisfaction Possible. The one exception here being if the user could have found that same information on the search results page before clicking on the ad. If that is the case, the landing page does not deserve a positive rating.
Do not give a landing page a Satisfaction Possible or Satisfaction Likely rating if you do not trust the information found on that landing page or if you would not make a purchase from the advertiser site. A page that offers the exact product that a user is looking for is useless unless the user trusts it enough to actually make a purchase there. A seemingly trustworthy merchant selling a particular camera at a particular price might deserve a better rating than a page that clumsily aggregates a random set of products, even if the same camera at the same price is offered on that page too. Similarly, a page offering the exact information that the user is looking for is not useful if there is no reason to think that the information is correct. For example, if the user seeks some medical information, a site belonging to a medical school is a good source of trustworthy information while a blog post by an unknown person is a much more doubtful source. Never use a rating of Satisfaction Likely or Satisfaction Possible if the page appears scammy or harmful.
Specific Versus General: Mismatch Between Queries and Landing Pages
Sometimes when the query is for a specific product, the landing page is basically on target but much broader or much more specific than the query.
If the landing page has the product specified in the query, but a huge number of other products too, this may be a decent experience, but probably is not good enough to get into the Satisfaction Likely range in most cases.
If the query is for something general, like [ camera ], the landing page might be extremely specific. For example, a product page for a particular model of camera from a particular manufacturer with a particular set of options. In this case, too, it might appear to be a decent experience, but it probably is not good enough to get rated as Satisfaction Likely in most cases.
You may judge that in particular cases, the experience is better or worse than the guidelines above would suggest. For example, if the page has a huge number of different products, but the product in the query is clearly the most prominent and the first thing you see, you might decide it deserves Satisfaction Likely; if it’s so buried in the other products that you don’t even realize it’s there, you might decide it deserves a negative rating. Similarly, if the query looks general and the landing page is for a very specific product, you might think that the product is so obviously the best possible thing to offer for that query that it deserves Satisfaction Likely; on the other hand, if the product is technically in the right category but very very unlikely to be what the user wants (for example, an expensive antique camera requiring glass plates instead of film for the query [ camera ]), you might decide that it deserves a negative rating.
Distinguishing Between Dissatisfaction Possible and Dissatisfaction Likely
Dissatisfaction Likely and Dissatisfaction Possible are negative ratings. If you think that the user will have a negative experience, always use either Dissatisfaction Possible or Dissatisfaction Likely. If you have no particular reason to think a page will interest the user, always use either Dissatisfaction Possible or Dissatisfaction Likely.
- If the page is marginally related to the query and you think that there’s a small chance the user would be interested, use Dissatisfaction Possible.
- If the page can eventually lead to what the user wants, but only through many clicks or through clicks that lead to an entirely different website, use Dissatisfaction Possible.
- If the page offers something that you think the user might be interested in, but not what the user was looking for and not especially close to it, use Dissatisfaction Possible. For example, if the user is looking for baseball gloves, and the landing page offers athletic socks, there’s probably some chance that the user might be interested. However, it’s not what the user was looking for, and not all that close to it, so it deserves Dissatisfaction Possible.
- If the page can eventually give the user what he or she is looking for, but the process is protracted and difficult, use Dissatisfaction Possible.
- If the page has nothing to do with the query, use Dissatisfaction Likely.
- If the query is for a product or service, and neither the product/service nor anything close to it can be purchased from the page, use Dissatisfaction Likely.
- If the query or a word in the query has two meanings, it is clear which meaning is intended by the user, and the advertiser responds to the wrong meaning, use Dissatisfaction Likely. For example, [ cars 2 ] refers to a movie. A page for a car dealership is clearly a bad landing page for this query, even if it might be a good result for
- [ car sales ].
- If the page looks like a scam, you think users could be harmed by it, or it either attempts to trick the user into downloading something by labeling a download button in a confusing way or tries to download a file without action by the user, use Dissatisfaction Likely.
- If the page loads but is completely unusable (for example, because some content does not load, or page doesn’t display properly) use Dissatisfaction Likely. If enough of the page does not load at all (for example, you encounter a 404 error), use the Error Did Not Load flag instead of a rating.
- If the page is very bad for any other reason, use Dissatisfaction Likely.
Use these flags to indicate that a query is unrateable. This means that it, and the LP paired with it, are not eligible to be assigned ratings. A Search Ads query is unrateable if it has one of the following problems:
- it is in a language other than the task language (Foreign Language)
- it is unambiguously pornographic or about sexual services (Porn)
- it is complete nonsense; research reveals no plausible meaning (Nonsense)
- it was transcribed incorrectly, using an English rather than Cyrillic keyboard for Russian words (Russian Transcription Error)
If you use one of these flags, all of the later questions will turn gray and don’t need to be answered.
Foreign Language Query
Use this flag when the query is in a language other than the language of the task. If the query contains words or phrases in another language, but there is enough content in the task’s language that it is understandable, do not use this rating. If the query appears to be in a foreign language, but research reveals that the query term may be commonly used in your rating language or is the name of a specific group, product, or person, do not use this rating.
Remember to check the language of the task, especially if you work in multiple languages. If your rating language is English, you rate ads in English for English queries. If you rate in another language, you will rate some tasks in that language and some tasks in English. Your rating language is always designated at the top of the task page. Even if you speak the language of the query, if the task is supposed to be for a different language, use this rating.
Use this flag only for queries that are unambiguously for pornographic content or sexual services. Queries for racy or suggestive content, medical information, or art photos generally shouldn’t get this rating. Queries for dating services generally shouldn’t get this rating, unless those dating services depict nudity or specifically identify themselves as sexual rendezvous services.
Use this flag for queries that are complete nonsense, where research reveals no plausible meaning. As you research, take into consideration that queries that may look like nonsense might actually turn out to be meaningful. The following are examples of queries that do have meaning and should not receive the Nonsense Query flag:
- a misspelling
- a product code or model number
- technical specifications
- a partial web address or YouTube video ID
- a specific username or Twitter handle
- an uncommon acronym or abbreviation
Don’t assume that a query is nonsensical just because you do not immediately know what it means. Encountering a completely nonsensical query is rare. Most queries mean something, so you should always research the query, even if at first it seems like nonsense. Only use this rating when there is no way for you to reasonably guess about user intent, even after researching the query.
Russian Transcription Error
This flag applies only to raters working in Russian. If you are working in a language other than Russian, this flag will never be applicable to your tasks, and you should not use it. If you are working in Russian, you will receive separate instructions for determining when queries should be considered transcription errors.
While you will not be able to assign LP ratings after using one of these Query Flags, you will still need to submit the task for your answers to be recorded.
Landing Page Flags
If a Landing Page meets the criteria for using one of the following flags, please use that flag. If criteria are not met for a flag, do not use the flag.
Use the Navigational Bullseye flag when both these things are true:
1. The query appears to be a search for a particular website.
2. The landing page is that site.
Not every query is a search for a particular website–in fact, the vast majority are not.
Use this flag when the landing page is in a language other than the language of the task, with no obvious way of getting to a version in the language of the task. Remember to check the language of the task, especially if you work in multiple languages. Even if you speak the language of the page, if the task is supposed to be for a different language, use this flag.
Don’t use this flag if there is some clear way to get to a version in the target language. For example, if you are rating a Japanese task, a landing page in English with a Japanese flag in the corner pointing to a Japanese version of the site should not get this flag.
If you use this flag, some of the later questions will turn gray and don’t need to be answered.
Use this flag when both these things are true:
1. The query is not a search for pornographic content or sexual services. If the query has both a pornographic interpretation and a non-pornographic interpretation, assume that the non-pornographic interpretation is the actual user intent.
2. The landing page offers pornographic content or sexual services.
Use this flag only for unambiguously pornographic content or sexual services. Racy content with no nudity, nudity in a medical context, or art photos generally shouldn’t get this flag. Dating services generally don’t get this flag unless they depict nudity or specifically identify themselves as sexual rendezvous services. A page with racy content, nudity in an art or medical context, or dating services may deserve a negative rating if it doesn’t match what the query appears to be looking for, but it shouldn’t get the flag.
If you use this flag, some of the later questions will turn gray and don’t need to be answered.
Use this flag when any of the following happens:
1. Clicking on the Visit Landing Page button initiates an attempt to download a file.
2. Some link, button, or graphic on the landing page initiates a download when clicked, but does not clearly indicate that it will do so. For example, a big red button that says “Enter site” or “Check the weather,” but starts a download when clicked, deserves the flag. A similar button that says “Get It Now” or “Click here to download” does not.
Never install downloads that a site tries to initiate in this way: it is not part of the rating process.
If you use this flag, some of the later questions will turn gray and don’t need to be answered.
Error / Did Not Load
Your job when evaluating a search ads task is to evaluate content provided by the advertiser (the ad creative and landing page). Use the Error/Did Not Load (EDNL) flag to indicate that you cannot evaluate the landing page because there is no landing site content provided by the advertiser. There are several reasons why you might not be able to access landing site content provided by the advertiser, including:
- the page or site no longer exist
- the page or site are under construction
- your browser is not able to find or access the page we provided you
- your virus/malware protection software blocks you from accessing the site
- the landing page opens using a 3rd-party program (such as iTunes) that you do not have installed
It’s not always easy to immediately determine if the EDNL flag should be used because different things can happen when a landing page is not working properly. Here are some examples of what you might see when no landing site content is available to evaluate:
- a completely blank page
- a generic Not Found message generated by your web browser
- a generic error message generated by the advertiser’s server
- a generic webpage (often filled with affiliate links) shown by the hosting service in place of the actual landing page
- a search results page shown by your internet service provider because the actual landing page cannot be accessed
- a notice that the site or page is under construction with no way to access any other part of the landing site
In all these cases you should use the EDNL flag because you cannot access any content from the actual landing site to evaluate. In the first two examples, above, there is little or no content to evaluate. In the last four examples, there may be content you can see, but it is either not content from the landing page advertiser (e.g. the hosting service, browser, ISP), or the entire advertiser site is inaccessible.
Note that a landing page could have an error on it but still have landing site content or a way to access landing site content on the page. Here are some examples of things you might see when there is an error on the page but advertiser content is still available to evaluate:
- a page which partially loads
- an error saying that the page could not be found, but linking to another part of the landing site
- an error stating that the product could not be found, but page provides alternatives or a way to search the landing site for other products
- a blank page or an error page that still has site navigation tools (usually on the top or side)
- an error page which automatically redirects to and loads a working page on the landing page advertiser site
- a landing page which is blocked by a registration form
If an advertiser landing page provides enough content to rate, don’t use the EDNL flag. In the cases above, the flag is not used because there is at least some advertiser content on the LP upon which to base your evaluation on.
If you use this flag, some of the later questions will turn gray and won’t need to be answered.
Secondary Interpretation of Query
Use this flag when the landing page content indicates that the advertiser is targeting a clearly secondary interpretation of the query. An interpretation is secondary if it’s reasonable, but there is some other interpretation of the query that you consider much more likely. Don’t use this flag with interpretations that are wrong or unreasonable. Don’t use this flag if you think that the query has multiple, equally likely meanings, and the advertiser is targeting one of those meanings. Do use the flag where the query has multiple, equally likely meanings and the advertiser targets an obscure or less-likely meaning. Please review the main guidelines for instructions on how to approach the scale rating when you use this flag.
If you have questions about this project that are not answered by the instructions above, please review the Rater Hub, which contains additional content about Search Ads rating. If you encounter a technical problem with this rating task, use the “Report A Problem” link in the lower-right hand corner of the rating page.