Work From Home
Close Match Evaluation Instructions – LionBridge – Work At Home
In this project, you will be looking at keywords and queries, making judgments about how their intents are related, and deciding how close in meaning the keywords are to the queries.
What are keywords and queries?
You already know what queries are: a user query is the set of words that a user enters into the Google search engine. User intent is what the user hopes to accomplish by using the Google search engine.
Keywords are words and phrases that advertisers choose in order to have their advertisements to show up alongside particular Google search results. Despite the name, keywords don’t have to be single words. Most of the keywords you encounter in this project will contain multiple words. Advertiser intent is what the advertiser hopes to accomplish by placing their ad alongside a query that matches the keyword they’ve chosen. Advertisers often formulate keywords very similarly to how users enter search queries, and expect their advertisements to show on queries that have a very similar meaning to their specified keyword.
You should research the keyword and query to form an opinion about what the advertiser and user meant by them, and what each may have been looking for.
Task Description
In each task, you will be presented with a keyword and a query. You will be asked to decide what these terms mean, and how their user and advertiser intents are related to each other. We want to know whether advertisements that are relevant to the keyword will also be relevant to the query.
Before you begin, research each term by clicking on it to look at the Google search results page for that term. If a keyword or query has several plausible meanings, focus on the one where the intent makes the most sense within the context of the keyword-query pair as you answer the questions in the task.
When you are ready, you will be asked to define how the intent of the keyword and the intent of the query are related.
If you are not able to answer this question because the keyword or query is unclear, ungrammatical, missing or in a foreign language, select the ‘unrateable’ checkbox next to the keyword and query. You will then be asked to identify whether the problem lies with the keyword, the query, or both.
How are these two terms related?
Your task is to determine how the keyword and query are related. There are five options:
| Rating | Rating Description |
|---|---|
| Same intent | The keyword and query have the same intent. They refer to the same product, service or concept possibly using synonyms, paraphrases, or implied terms. |
| Query more specific | The query is a more specific case of the keyword. The query adds a more specific, non-implied, aspect, product sub-type, location, or brand, but preserves the full intent of the keyword. Nothing is more specific in the keyword. |
| Keyword more specific | The keyword is a more specific case of the query. Some aspect of the keyword is not reflected in the query: the query may be missing a non-implied brand, location, or product specification that is present in the keyword. Nothing is more specific in the query. |
| Related but different intents | The keyword and the query have related but different intents. For example, they are about related but different products. The query is at the same time more and less specific than the keyword. The query refers to an accessory or service for the keyword (or the other way around). |
| Unrelated intents | The keyword and the query have completely unrelated intents. For example, they represent items from completely different non-overlapping product areas. |
Below you will find additional guidance and specific examples that will help you correctly categorize tricky and unclear cases.
Option 1: Same intent
Select this option if the keyword and the query have the same intent. For this task, this does not mean that the terms need to contain all the same words or be phrased in the same way: as long at the keyword and query refer to the same product, service, or concept they should be rated as having the same intent.
You should pick this option if you think the keyword and query pick out the same things as depicted in the picture below. The query, fruit, and the keyword, fruit, pick out the same set of items.
Not every pair you rate will be as easy as the pair above. Below we review several common types of Query-Keyword pairs that have the same intent. Sometimes you will encounter a mix of the cases covered below. In this case, you should still select this option.
1) Synonyms: the keyword and query express the same intent using different words, for example, synonyms or paraphrases.
| Keyword | Query | Explanation |
|---|---|---|
| safety shoes | hazard shoes | The query may use synonymous words to refer to the same. |
| pediatrician | doctors for children | The query is a paraphrase of the keyword. |
| diaper bag for a man | men diaper bag | The keyword paraphrases the query with additional words. |
| climate change | global warming | The keyword and query are synonyms; they use different words but target the same concept. |
2) Identical Search Intent: the keyword and query target the same information and express essentially the same search intent.
In the table below are some examples of keywords and queries that have very similar search intents. These keywords and queries target the same information even though they are not exact paraphrases of each other.
| Keyword | Query |
|---|---|
| male infertility low sperm count | how to get pregnant with low sperm count |
| romantic music online | love songs online playlist |
| global map turkey | turkey’s location on the world map |
Note: You can usually tell that the keyword and query express the same search intent by checking the actual search results for each. However, you should be careful since sometimes queries and keywords that have different search intents show similar search results. For example, ‘flights la ny’ shows similar results to ‘flights ny la’ even though the search intents are different.
3) Implied Terms: the keyword contains an additional word or words that are always or almost always implied by other terms in the query (or the other way around).
In the table below you can see keywords or queries that should be rated as having the same intent even though the keyword or query has additional words that are not included in the other.
| Keyword | Query | Explanation |
|---|---|---|
| toyota camry | camry | The Toyota Camry is a specific car model. It is fine to drop the brand “toyota” here since it is implied by the model name “camry”. |
| houses for rent in Colville | houses for rent in Colville wa | Since Colville is in WA (Washington State), “wa” is implied by Colville. |
| daydream headset | daydream vr headset | Since all daydream headsets are vr (virtual reality) headsets, the term “vr” is implied in the keyword. |
4) Implied Intent: the keyword has the same implied intent as specified by the query (or the other way around).
A keyword or query has an implied intent if a user searching for a concept does not need to specify that intent to achieve his or her goal.
| Keyword | Query | Explanation |
|---|---|---|
| hydrographics | hydrographics how it works | Since “hydrographics” is a complex topic, the query, which asks for an explanation of hydrographics, should be treated as having same intent as the keyword. |
| what is health insurance marketplace | health insurance marketplace | A user searching for “health insurance marketplace” probably wants to find out details about it, so the keyword which offers a definition should be treated as expressing the same intent. |
| vacation home rentals | vacation homes | Since vacation homes are almost always rented, the keyword and query have the same intent. |
| play games online | games online | A user looking for “games online” likely wants to play games online, so the keyword and query have the same intent. |
| christmas gifts | christmas gift ideas | When a user searches for gifts (in this case ‘christmas gifts’), they are likely to want gift ideas, so this query has the same intent as the keyword. |
5) Commercial Intent: the query expresses the intent to purchase the product or service referred to by the keyword (or the other way around).
| Keyword | Query |
|---|---|
| pro tools 11 download | buy pro tools 11 download |
| where to purchase rx bars | rx bars |
| hyundai accent | hyundai accent price |
| adidas shop | adidas |
Note: if the keyword or query expresses a specific price point, for example “cheap” or “discount”, that is not present in the other you should not select this option. For example the keyword or query “handbags” includes expensive handbags like Louis Vuitton handbags, but the keyword or query “discount handbags” would exclude expensive handbags. Therefore, you should treat “discount handbags” as more specific than “handbags”.
Examples that don’t fall into this option (same intent).
Below you will find some additional tricky examples that seem like they might belong in this category, but don’t.
| Keyword | Query | Correct Rating | Explanation |
|---|---|---|---|
| acer laptop | used acer laptop | The query is more specific than the keyword. | Although the addition of the word ‘used’ implies commercial intent, it also implies that the user does not seek new laptops, which is included in the keyword. |
| train from tahoe to sj | train sj to tahoe | The keyword and query are related. | Although both the keyword and query both refer to trains between Tahoe and sj (San Jose), the keyword and query have different directions of travel. |
| iphone case | iphone with case | The keyword and query are related. | The keyword and query are about different products: the keyword is about iphone cases while the query is primarily about iphones (that come with cases). |
Option 2: Query more specific
This option applies if the query adds a more specific aspect (for example, a product sub-type, location, or brand) to the keyword, but preserves the main intent of the keyword.
You should pick this option if you think the query picks out a narrower set of things than the keyword as depicted in the picture below. The query, apples, picks out a narrow set of items than the keyword, fruit.
Again, the case depicted above is simpler than many cases you will see. Below we review several common types of queries and keyword that should fall in this category.
1) Product Subtype: the query is a specific product subtype of the keyword.
| Keyword | Query | Explanation |
|---|---|---|
| office furniture | office desk | Since an office desk is a type of office furniture, the query is more specific than the keyword. |
| shoes | women’s boots | “women’s boots” belongs to the category “shoes”. |
2) Additional Aspect: the query has an additional non-implied aspect to its intent, but still preserves the main intent of the keyword.
| Keyword | Query |
|---|---|
| aftermarket auto parts | cheap after market auto parts |
| wineries in long island | best long island vineyards |
| lunch recipes | easy lunch recipes |
| buy printer | color printer |
| micro sd | 64gb micro sd card |
| coffee beans | buying coffee beans online |
| play games | play games for free |
3) Specific Product Intent: the query adds specific intent directly related to the product or service referred to by the keyword.
Common additional intents include users looking for reviews, comparisons, coupons, or discounts on a specific product or service.
| Keyword | Query |
|---|---|
| hp notebook | hp notebook reviews |
| credit cards | credit card comparisons |
| oil change | oil change coupon |
Note: if the keyword describes a product and the query intent shifts from the product to a service or accessory for that product, you should treat them as having related but different intents (option 4). For example ,”iphone” and “iphone case” (accessory), “tesla” and “tesla charging station” (service), and “laptop” and “laptop scratched screen” (problem) are related but have different intents.
4) Specific Product Information: the query requests specific information about the properties of the product, service or topic that the keyword refers to.
| Keyword | Query | Explanation |
|---|---|---|
| learn piano | how long does it take to learn the piano | The user query asks for specific information about the aspect of the service offered by the keyword. |
| hp notebook | hp notebook screen size | The keyword is about a specific product, and the query requests information about a specific aspect of that product. |
Note: if the keyword describes a product or service and the information requested by the query shifts the intent from the product or service to a problem with that product or service, you should treat them as having related but different intents (option 4). For example, an advertiser keyword “hp notebook” intends to offer “hp notebooks” while a user query “hp notebook won’t turn on” requests help with a specific problem with an “hp notebook”. The problem the user has is related to product, but they are distinct enough that they should be treated as related.
Similarly, if the query shifts the intent from a product to an accessory or service related to the product, you should treat them as having related but different intents (option 4). For example, if the keyword is “hp notebook” and the query is “hp notebook power cord” (an accessory) or “hp notebook repair” (a related service) , you should mark the keyword and query as having related but different intents.
5) Request Additional Information: the query formulates a specific request or question about the problem, topic, or question that the keyword refers to.
If the keyword offers a general problem, topic, or question and the query asks a specific question or makes a specific request about the keyword, you should select this option.
| Keyword | Query |
|---|---|
| child custody | do i have enough to gain full child custody |
| puppy fleas | what do you do for fleas on a 2 week old puppy |
Note: this is only applicable if the keyword describes a problem, topic, or question. If the keyword describes a product or service, this does not apply.
6) Specific Location: the query adds a location to the keyword or specifies a more specific location than the keyword.
| Keyword | Query | Explanation |
|---|---|---|
| doctors | doctors near me | The query specifies doctors in a location while the keyword is for doctors generally. |
| texas apartments | dallas apartments | Since Dallas is a city in Texas, the query is asking for a more specific location than the keyword. |
7) Specific Brand: the query adds a specific brand, business, or merchant to the keyword or specifies a more specific brand, business, or merchant than the keyword.
Note: You should treat a query for a brand, business, or merchant to be more specific than the a keyword for the main product or service the brand, business, or merchant is primarily associated with. However, if a brand or merchant is associated with many types of products (for example, Amazon is associated with a wide range of products), you should not consider it as more specific than any of its particular products (for example, keyword ‘books’).
| Keyword | Query | Explanation |
|---|---|---|
| printer repair | epson printer fix | The query specifies the brand of printer to fix. |
| general motors cars | chevrolet cars | Chevrolet is a specific division of General Motors. |
| auto parts | rockauto | The query is about the merchant ‘rockauto’ that mainly sells autoparts. |
Examples that don’t fall into this option (query more specific).
Below you will find some additional tricky examples that seem like they might belong in this category, but don’t.
| Keyword | Query | Correct Rating | Explanation |
|---|---|---|---|
| set up | set up chromecast | The keyword is unrateable. | Although one could complete the keyword with “chromecast” to get the query, there is no way to know that the advertiser intended this. We can’t know what the advertiser meant because the keyword is incomplete. |
| cast iron | cast iron repair | The keyword and query are related. | Although, the query only adds words to the keyword, the query shifts the intent from a product (cast iron) to a service (repair) related to the product. |
| 2015 truck | 2015 truck for sale | The keyword and query have the same intent. | Although the query adds words to the keyword, the additional words only specify a commercial intent, which can be assumed in the keyword. |
| laundry basket | target | The keyword and query have related but different intents. | The query “target” picks out a merchant that sells many products, including laundry baskets. However, since Target does not primarily sell laundry baskets, you should chose Related. |
Option 3: Keyword more specific
This option applies if an aspect of the keyword is not reflected in the query. The missing aspect may be a brand, a location, a product specification or other additional information.
You should select this option if the query picks out a broader set of things than the keyword as depicted in the picture below. The keyword, apples, picks out fewer items than the query, fruit.
When rating, you should consider the same aspects of the keyword and query that you would take into account when deciding whether the query is more specific than the keyword (option 2) above.
If the keyword does not preserve the main intent of the query, do not choose this option: instead mark the pair as having related but different intents (Option 4) or unrelated intents (Option 5).
Option 4: Related but different intents
This is the case for example if the keyword and query are about related but different products, or if the query is at the same time more and less specific than the keyword.
One way for a keyword and query to be related is if they pick out overlapping sets of items. For example, the query, cats, and keyword, happy animals, pick out different but overlapping items in the picture below.
Another way a keyword and query can be related is if they pick out entirely different sets of items that still have something to do with one another. For instance the query, orange juice, and the keyword, orange, should be treated as related because orange juice is made from oranges.
Below we review several types of queries and keywords that should fall into this category.
1) Different Products: the keyword and query specify related but different products, services, problems, or topics.
| Keyword | Query | Explanation |
|---|---|---|
| macbook pro | macbook air | Keyword and query are different models of the same brand. |
| large format printing | used wide format printers | The keyword asks for the service “printing” while the query asks for the product “printer”. |
| sofa recliner | super size recliner | One product is a sofa, the other a chair. |
2) Accessory/Service Shift: the keyword shifts the intent expressed in the query, for example, the keyword refers to an accessory or a service for the query’s product (or the other way around).
| Keyword | Query | Explanation |
|---|---|---|
| macbook pro | macbook pro keyboard | The query is an accessory for the product referred to by the keyword, so these have related but different intents. |
| toyota car parts | toyota | The query “toyota” implies that the user is looking for toyota cars. The keyword “toyota car parts” picks out accessories for Toyota cars. |
| desktop pc repair | desktop pc | The keyword refers to a service related to the product referred to by the query, so these have related but have different intents. |
| hairdressers in nyc | training in hairdressing nyc | The keyword offers the a hairdressing service in NYC (New York City). The query is for a related service, namely training to become a hairdresser in NYC. |
| granite countertops | granite countertops images | The keyword refers to granite countertops while the query specifies that the user is looking for images of granite countertops. The intents are related but different. |
Note: If the query shows that the user’s main intent is focused on the product in the keyword itself, then the query is more specific than the keyword (option 2). For example, if the user is looking for reviews of a product or service, wants to compare the product or service to other alternatives, or is looking for coupons, discounts, or delivery of the product, the query is more specific than the keyword.
3) Incompatible Aspects: the keyword and query refer to the same product, service, problem, or topic but include incompatible locations, brands, merchants, or other properties.
| Keyword | Query |
|---|---|
| italian restaurants sf | italian restaurants LA |
| nike shoes | reebok shoes |
| refugee crisis solutions | refugee crisis causes |
4) More and Less Specific: the keyword is at the same time more and less specific than the query.
| Keyword | Query | Comment |
|---|---|---|
| arthritis of the cervical spine | arthritis of the spine surgery | “cervical” is not reflected in the query. “surgery” is not reflected in the keyword. |
| first aid certification | first aid children | “children” is not reflected in the keyword. “certification” is not reflected in the query. |
| used campers for sale | campers for sale in virginia | The keyword offers used campers that may or may not be in Virginia, while the query is for campers in Virginia that may or may not be used. |
5) Direction Switching: the keyword and query refer to the same action, but switch directions.
The instructions for Option 1 (same intent) discussed cases in which the keyword and query can use different words or different orders of words to express the same intent, but sometimes you will encounter keywords and queries that use the same or similar words, but express different intents. This is often the case when the order of the words in the keyword or query imply a direction.
| Keyword | Query | Explanation |
|---|---|---|
| turkey to toronto flight | toronto turkey flight | Both the keyword and query are looking for flights between Toronto and Turkey, the keyword picks out flights from Turkey to Toronto, but the query picks out flights from Toronto to Turkey, so the intents are different. |
| rupee to dollar rate today | today’s dollar rate in rupees | The keyword and query specify different directions of the exchange rate between Rupees and Dollars. |
| translate french to german | translation german french | The keyword and query specify different directions for the translation between French and German. |
Option 5: Unrelated intents
Select this option if the keyword and query have completely unrelated intents.
Select this option if the keyword and query represent non-overlapping items that come from entirely different non-overlapping domains or product areas. For example, the query, orange, and the keyword, orange cat, should be considered unrelated since they have no items in common, and the items they pick out don’t really have anything to do with one another.
Below are some examples of keywords and queries that are unrelated.
| Keyword | Query |
|---|---|
| lord of the rings | wedding rings |
| las vegas entertainment options | stock option quotes |
| international year of the potato | the international |
Note: If the keyword and query have partially overlapping intent, you should choose the option related but different instead. For example, the query “1984 symbolism essay” is related to the keyword “1984 book”, because they are both related to the novel 1984.
Advice for ambiguous queries and keywords
1. Start by clicking on the keyword or query and looking at the search results for each. These may reveal the correct interpretation of a keyword or query.
2. If a keyword or query is unclear, has many interpretations, expresses only partial concepts, or includes non-sensical or ungrammatical text (usually keywords), that you cannot fully interpret you should treat the keyword or query as unrateable. Some examples include keywords or queries like “blue”, “cakes in”, and “inthe word”.
3. If a keyword or query has several reasonable and clear meanings that are all equally likely you should consider interpretation of the keyword or query that is mostly likely and best matches the other. Below are some examples. However, try not to stretch the meaning of the keyword or query or look for one that isn’t clearly present.
| Keyword | Query | Rating |
|---|---|---|
| luxury cars | jaguar | Since Jaguar is a brand of luxury cars, the query is more specific than the keyword. |
| big cats | jaguar | Since jaguars are big cats, the query is more specific than the keyword. |
| contact lenses | contacts | Since contact lenses are often called ‘contacts’, the keyword and query have the same intent. |
| address book | contacts | Since contacts can also refer to a list of people who’s contact information you have, the keyword and the query have the same intent. |
4. If a keyword or query has several reasonable interpretations and one is clearly more plausible than the others, you should consider only that interpretation. For example, the keyword or query [paris] could refer to plaster of paris or Paris, France. Because the interpretation Paris, France is much more plausible, you should provide a rating as if Paris, France were the only interpretation.
The Unrateable Flag
If either the keyword, the query, or both are unclear, nonsensical, missing, or in a foreign language, select the unrateable flag. If you select the unrateable flag, you will be asked to specify whether the problem is with the keyword, the query, or both. Choose the appropriate option and submit the task.
Note: If the keyword or query is in a foreign language, you should select the unrateable flag only if the keyword or query is not something you would expect a typical speaker in the locale to understand. For example, a query that was a mix of German and English should not be marked unrateable in a German language task, since many people who primarily speak German also know at least some English.
Note: Porn-related terms are expected for this task, and should be rated normally.
If either the keyword or the query is so ambiguous or so general that it is difficult to understand its intent, it may not be rateable. If it’s not clear what the advertiser wanted when they chose a keyword, or what the user was looking for when they entered a query, it may be impossible to provide a rating. Below are some examples of keywords or queries that you should rate as unclear or nonsensical.
| Keyword or Query | Explanation |
|---|---|
| take | The keyword or query “take” could refer to anything from ‘taking pills’ to ‘take a job’; it’s too hard to rate this keyword/query because we cannot say what the advertiser or user really wanted. |
| blue | The keyword/query “blue” is too general for have a clear intent. The user or advertiser could be interested in anything from blue dresses to the blue man group. |
| cakes in | The keyword or query “cakes in” is nonsensical. |
| banda larga | The keyword or query is in Spanish while English ratings were requested. |
Keep in mind that not all keywords or queries that seem nonsensical actually are. It is important to research the keyword and query to make sure that you are not misunderstanding its intent.
If you encounter a technical problem with this rating task, use the “Report A Problem” link in the lower-right hand corner of the rating page.
