Rater Behavior – Correct Use of Flags – Lionbridge

The purpose of this course is to provide you with training on when to correctly assign flags when rating tasks. It is important for raters to master the use of flags as failure to use them correctly can lead to removal from the program.

As a general rule – you should always try to rate a task if you can.  If you cannot, then please look to the unrateable or a more specific flag to signal the issue you encounter.

Introduction :

What are Flags?

Flags are signals you can apply to indicate that something in the task is unexpected.  It is very important that you review instructions and text associated with each flag to ensure you are assigning it correctly.

You should always check to see if a flag can be applied to a problematic task before you report a problem.  You will often find reference to flags in the task instructions, be sure to check.

Please note that misuse of flags can result in your account being suspended.  For this reason it is very important that you use them correctly.

Above is a sample of a flag that should be assigned when the users query cannot be understood. 

Examples of Flags:

Unexpected Porn

Unexpected Download

Error Did Not Load (EDNL)

Ad Did Not Load

Video Did Not Load or Play

Image Failed to Load

Foreign Language

Wrong Language

Other

Unrateable

Incomprehensible

Nonsense Query

Why use Flags?

Why are there flags?

Flags are signals you can apply to indicate that something in the task is unexpected, as per task outline / instructions. 

You should always try your best to complete a task.  For example, if a video in a task is in a language you are not familiar with, but you can still attempt the task – then do!!  The process should be as below:

1. Attempt the task
2. Assign a flag
3. Report a problem

It is very important to keep in mind, that flags are different depending on the task you are completing.  Some flags may even be similar in name but applied in a different way. For this reason it is very important to take your time when considering the use of a flag.

Some flags might only be applicable in a specific task type or for a specific scenario.

Always check if you can apply a flag before you report a problem with a task. 

“Unexpected Porn” Flag

The Flag “Unexpected Porn”  

This flag can be found in several different task types. 

The key word is “Unexpected”.

We assign this flag only if the result (for example: Ad Creative, Landing Page) contains porn but the query is NOTasking for it. This is for clear and unambiguous porn content only.

Example: The query [Walmart] should NOT generate porn results. If the responding AC and/or LP contains porn, please do assign the “Unexpected Porn” flag. 

If however the query is asking is seeking porn and the AC /LP are interpreting the intent correctly and display expected porn, then do not use this flag. 

Error/Did Not Load (EDNL) Flag

This is a very common flag and can be found in several task types.

Unless the instructions for the task say otherwise:

• We assign this flag if the result (for example: Ad Creative, Landing Page) does NOT load at all, or that you can’t see enough to evaluate the result correctly.

This flag reports that an Ad Creative, Landing Page or other elements of the task that should be evaluated, can not be evaluated at all. 

If the Ad Creative does not load

If the Landing Page gives a generic error message (i.e. Error 404)

The Landing Page is completely blank

Unrateable Flag

The unrateable flag appears on tasks where there is an issue that prevents you from submitting accurate ratings.  You may find that tasks have more specific flags, in this case please use these.  

The layout of flags differs depending on the task – below are a some samples to help you become familiar.

‘Foreign Language’ Flag

Too much foreign language to rate.

An important concept to understand is the use of the Foreign Language flag.  When completing tasks you may encounter tasks that contain foreign language.  In cases such as this you should first consider if it is possible to complete the task.  Many tasks can be completed without understanding the language – for example if an Ad contains inappropriate content.

At all times refer to the instructions – but if you encounter a task where there is too much foreign language to rate then please flag as such.

Below are some samples of the Foreign Language flag: