Building Web Reputation Systems- P20 pps

Customer care could be removed from the loop—in most cases—by shifting the content removal process into the application and giving it to the users, who were already the source of the abu

• Some community members would report abuse for altruistic reasons: out of a desire to keep the community clean (See the section “Altruistic or sharing incen-tives” on page 113.) Downplaying the contributions of such users would be critical; the more public their deeds became, the less likely they would continue acting out

of sheer altruism

• Some community members had egocentric motivations for reporting abuse The team appealed to those motivations by giving those users an increasingly greater voice in the community

The High-Level Project Model

The team devised this plan for the new model: a reputation model would sit between the two existing systems—a report mechanism that permitted any user on Yahoo! An-swers to flag any other user’s contribution and the (human) customer care system that acted on those reports (See Figure 10-3.)

This approach was based on two insights:

1 Customer care could be removed from the loop—in most cases—by shifting the content removal process into the application and giving it to the users, who were already the source of the abuse reports, and then optimizing it to cut the amount

of time and offensive posting by 90%

2 Customer care could then handle just the exceptions—undoing the removal of content mistakenly identified as abusive At the time, such false positives made up 10% of all content removal Even if the exception rate stayed the same, customer care costs would decrease by 90%

The team would accomplish item 1, removing customer care from the loop, by imple-menting a new way to remove content from the site—“hiding.” Hiding involved trust-ing the community members themselves to vote to hide the abusive content The reputation platform would manage the details of the voting mechanism and any related karma Because this design required no external authority to remove abusive content from view, it was probably the fastest way to cut display time for abusive content

As for item 2, dealing with exceptions, the team devised an ingenious mechanism—an appeals process In the new system, when the community voted to hide a user’s content, the system sent the author an email explaining why, with an invitation to appeal the decision Customer care would get involved only if the user appealed The team pre-dicted that this process would limit abuse of the ability to hide content; it would provide

an opportunity to inform users about how to use the feature; and, because trolls often don’t give valid email addresses when registering an account, they would simply be unable to appeal because they’d never receive the notices

Figure 10-3 The system would use reputation as a basis for hiding abusive content, leaving staff to handle only appeals.

Most of the rest of this chapter details the reputation model designated by the Hide Content? diamond in Figure 10-3 See the patent application for more details about the other (nonreputation) portions of the diagram, such as the Notify Author and Appeals process boxes

Yahoo! has applied for a patent on this reputation model, and that

ap-plication has been published: Trust Based Moderation—Inventors: Ori

Zaltzman and Quy Dinh Le Please consider the patent if you are even

thinking about copying this design.

We are grateful to both the Yahoo! Answers and the reputation product

teams for sharing their design insights and their continued assistance in

preparing this case study.

Objects, Inputs, Scope, and Mechanism

Yahoo! Answers was already a well-established service at the time that the community content moderation model was being designed, with all of the objects and most of the available inputs already well defined The final model includes dozens of inputs to more than a dozen processes Out of respect for intellectual property and the need for brevity,

we have not detailed every object and input here But, thanks to the Yahoo! Answers team’s willingness to share, we’re able to provide an accurate overall picture of the reputation system and its application

The Objects

Here are the objects of interest for designing a community-powered content moderation system:

User contributions

User contributions are the objects that users make by either adding or evaluating content:

Questions

Arriving at a rate of almost 100 per minute, questions are the starting point of all Yahoo! Answers activity New questions are displayed on the home page and on category pages

Answers

Answers arrive 6 to 10 times faster than questions and make up the bulk of the reputable entities in the application All answers are associated with a single question and are displayed in chronological order, oldest first

Ratings

After a user makes several contributions, the application encourages the user

to rate answers with a simple thumb-up or thumb-down vote The author of the question is also allowed to select the best answer and give it a rating on a 5-star scale If the question author does not select a best answer in the allotted time, the community vote is used to determine the best answer

Users may also mark a question with a star, indicating that the question is a favorite

Each of these rating schemes already existed at the time the community content moderation system was designed, so for each scheme, the inputs and outputs were both available for the designers’ consideration

Users

All users in this application have two data records that can hold and supply infor-mation for reputation calculations: an all-Yahoo! global user record, which in-cludes fields for items such as registration data and connection information, and

a record for Yahoo! Answers, which stores only application-specific fields Developing this model required considering at least two different classifications of users:

Authors

Authors create the items (questions and answers) that the community can moderate

Reporters

Reporters determine that an item (a question or an answer) breaks the rules and should be removed

Customer care staff

The customer care staff is the target of the model The goal is to reduce the staff’s participation in the content moderation process as much as possible but not to zero Any community content moderation process can be abused: trusted users may decide to abuse their power, or they may simply make a mistake Customer

care would still evaluate appeals in those cases, but the number of such cases would

be far less than the total number of abuses

Customer care agents also have a reputation—for accuracy—though it isn’t cal-culated by this model At the start of the Yahoo! Answers community content moderation project, the accuracy of a customer care agent’s evaluation of questions was about 90% That rate meant that 1 in 10 submissions was either incorrectly deleted or incorrectly allowed to remain on the site An important measure of the model’s effectiveness was whether users’ evaluations were more accurate than the staff’s

The design included two noteworthy documents, though they were not formal objects (that is, they neither provided input nor were reputable entities) The Yahoo! Terms of Service and the Yahoo! Answers Community Guidelines (Figure 10-4) are the written standards for questions and answers Users are supposed to apply these rules in eval-uating content

Figure 10-4 Yahoo! Answers Community Guidelines.

Limiting Scope

When a reputation model is introduced, users often are confused at first about what the reputation score means The design of the community content moderation model

for Yahoo! Answers is only intended to identify abusive content, not abusive users.

Remember that many reasons exist for removing content, and some content items are removed as a result of behaviors that authors are willing to change, if gently instructed

to do so

The inclusion of an appeals process in the application not only provides a way to catch false-positive classification by reporters, it also gives Yahoo! a chance to inform authors

of the requirements for participating in Yahoo! Answers, allowing users to learn more about expected behavior

An Evolving Model

Ideally, in designing a reputation system, you’d start with as comprehensive a list of potential inputs as possible In practice, when the Yahoo! Answers team was designing the community content moderation model, they used a more incremental approach

As the model evolved, the designers added more subtle objects and inputs Next, to illustrate an actual model development process, we’ll roughly follow the historical path

of the Yahoo! Answers design

Iteration 1: Abuse reporting

When you develop a reputation model, it’s good practice to start simple; focus only on the main objects, inputs, decisions, and uses Assume a universe in which the model works exactly as intended Don’t focus too much on performance or abuse at first; you’ll get to those issues in later iterations Trying to solve this kind of complex equation

in all dimensions simultaneously will just lead to confusion and impede your progress For the Yahoo! Answers community content moderation system, the designers started with a very basic model: abuse reports would accumulate against a content item, and when some threshold was reached, the item would be hidden This model, sometimes called “X-strikes-and-you’re-out,” is quite common in social web applications Craigslist is a well-known example

Despite the apparent complexity of the final application, the model’s simple core design remained unchanged: accumulated abuse reports automatically hide content Having that core design to keep in mind as the key goal helped eliminate complications in the design

From the beginning, the team planned for the primary input to the model to be

a user-generated abuse report explicitly about a content item (a question or an answer) This user interface device was the same one already in place for alerting customer care

to abuse Though many other inputs were possible, initially the team considered a model with abuse reports as the only input

Abuse reports (user input)

Users could report content that violated the community guidelines or the terms of service The user interface consisted of a button next to all questions and answers

Inputs.

The button was labeled with a flag icon, and sometimes the action of clicking the button was referred to as “flagging an item.” In the case of questions, the button label also included the phrase “Report Abuse.” The interface then led the user through a short series of pages to explain the process and narrow down the reason for the report

The abuse report was the only input in the first iteration of the model

At the core of the model was a simple, binary decision: should a content item that has just been reported as abusive be hidden? How does the model make the decision, and, if the result is positive, how should the application be notified?

In the first iteration, the model for this decision was “three strikes and you’re out.” (See Figure 10-5.) Abuse reports fed into a simple accumulator (see “Simple Accumula-tor” on page 48) Each report about a content item was given equal weight; all reports were added together and stored as AbusiveScore That score was sent on to a simple evaluator, which tested it against a threshold (3) and either terminated it (if the thresh-old had not been reached) or alerted the application to hide the item

Given that performance was a key requirement for this model, the abuse reports were delivered asynchronously, and the outgoing alert to the application used an application-level messaging system

This iteration of the model did not include karma

Figure 10-5 Iteration 1: A not-very-forgiving model Three strikes and your content is out!

This very simple model didn’t really meet the minimum requirement for the application—the fastest possible removal of abusive content Three strikes is often too many, but one or two is sometimes too few, giving too much power to bad actors The model’s main weakness was to give every abuse report equal weight By giving trusted users more power to hide content and giving unknown users or bad actors less power, the model could improve the speed and accuracy with which abusive content was removed

Mechanism and diagram.

Analysis.

The next iteration of the model introduced karma for reporters of abuse.

Iteration 2: Karma for abuse reporters

Ideally, the more abuse a user reports accurately, the greater the trust the system should place in that user’s reports In the second iteration of the model, shown in Fig-ure 10-6, when a trusted reporter flagged an item, it was hidden immediately Trusted reporters had proven, over time, that their motivations were pure, their comprehension

of community standards was good, and their word could be taken at face value Reports by users who had never previously reported an item, with unknown reputation, were all given equal weight, but it was significantly lower than reports by users with a positive history In this model, individual unknown reporters had less influence on any one content item, but the votes of different individuals could accrue quickly (At the same time, the individuals accrued their own reporting histories, so unknown reporters didn’t stay unknown for long.)

Though you might think that “bad” reporters (those whose reports were later over-turned on appeal) should have less say than unknown users, the model gave equal weight to reports from bad reporters and unknown reporters (See “Practitioner’s Tips: Negative Public Karma” on page 161.)

To the inputs from the previous iteration, the designers added three events re-lated to flagging questions and answers accurately:

Item hidden (moderation model feedback)

The system sent this input message when the reputation process determined that

a question or answer should be hidden, which represented that all users who re-ported the content item agreed that the item was in violation of either the TOS or the community guidelines

Appeal Result: Upheld (customer care input)

After the system hid an item, it contacted the content author via email and enabled the author to start an appeal process, requesting customer care staff to review the decision If a customer care agent determined that the content was appropriately hidden, the system sent the event Appeal Result: Upheld to the reputation model

Appeal Result: Overturned (customer care input)

If a customer care agent determined that the content was inappropriately hidden, the system displayed the content again and sent the event Appeal Result: Overturned to the reputation model for corrective adjustments.

The designers transformed the overly simple “strikes”-based model to account for a user’s abuse report history

The goals were to decrease the time required to hide abusive content, and reduce the risk of inexperienced or bad actors hiding content inappropriately

Inputs.

Mechanism and diagram.

The solution was to add AbuseReporter karma to record the user’s accuracy in hiding abusive content Use AbuseReporter to give greater weight to reports by users with a history of accurate abuse reporting

To accommodate the varying weight of abuse reports, the designers changed the cal-culation of AbusiveScore from strikes to a normalized value, where 0.0 represented no abuse information known and 1.0 represented the maximum abuse value The eval-uator now compared the AbusiveScore to a normalized value representing the certainty required before hiding an item

The designers added an AbuseReporter reputation claim, a normalized value, where 0.0 represented a user with no history of abuse reporting and 1.0 represented a user with

a completely accurate abuse reporting history A user with a perfect score of 1.0 could hide any item immediately

Figure 10-6 Iteration 2: A reporter’s record of good and bad reports now influences the weight of his opinion on other content items.

The inputs that increased AbuseReporter were Item Hidden and Appeal Result: Upheld The input Abuse Result: Overturned had a disproportionately large negative effect on AbuseReporter, providing an incentive for reporters not to use their power indiscriminately

Unlike the first process, the new version of the Content Item Abuse process did not treat each input the same way It read the reporter’s AbuseReporter karma, added a small constant to AbusiveScore (so that users with no karma made at least a small contribution to the result), and capped the result at the maximum If the result was 1.0, the system hid the item but, in addition to alerting the application, it updated the AbuseReporter karma for each user that flagged the item This reflected community consensus and, since the vast majority of hidden items would never be reviewed by customer care, was often the only opportunity the system had to reinforce the karma

of those users Very few appeals were anticipated given that trolls were known to give bogus email addresses when registering The incentives for both the legitimate authors and good abuse reporters discourage abusing the community moderation model The system sent appeal results messages asynchronously as part of the customer care application; the messages could come in at anytime After AbuseReporter was adjusted, the system did not attempt to update other AbusiveScores the reporter may have con-tributed to

The second iteration of the model did exactly what it was supposed to do: it allowed trusted reporters to hide abusive content immediately However, it ignored the

value of contributions by authors who might themselves be established, trusted mem-bers of the community As a result, a single mistaken abuse report against a top

con-tributor led to a higher appeal rate, which not only increased costs but generated bad feelings about the site Furthermore, even before the first iteration of the model had been implemented, trolls already had been using the abuse reporting mechanism to harass top contributors So in the second iteration, treating all authors equally allowed malicious users (trolls or even just rivals of top contributors) to take down the content

of top contributors with just a few puppet accounts

The designers found that the model needed to account for the understanding that in cases of alleged abuse, some authors always deserve a second opinion In addition, the designers knew that to hide content posted by casual regular users, the AbusiveScore required by the model should be lower—and for content by unknown authors, lower still

In other words, the model needed karma for author contributions

Iteration 3: Karma for authors

The third iteration of the model introduced QuestionAuthor karma and AnswerAuthor karma, which reflected the quality and quantity of author contributions The system compared AbusiveScore to those two reputations instead of a constant This change raised the threshold for hiding content for active, trusted authors and lowered the

Analysis.

threshold for unknown authors and authors known to have contributed abusive content

The new inputs to the model fell into two groups: inputs that indicated the quantity and community reputation of the questions and answers contributed by an author and evidence of any previous abusive contributions

Inputs contributing to positive reputation for a question

Numerous events could indicate that a question was valuable to the community When a reader took any of the following actions on a question, the author’s QuestionQuality reputation score increased:

• Added the question to his watch list

• Shared the question with a friend

• Gave the question a star (marked it as a favorite)

Inputs contributing to negative reputation for a question

When customer care staff deleted a question, the system set the author’s Question Quality reputation score to 0.0 and adjusted the author’s karma appropriately. Another negative input was the Junk Detector score, which acted as an initial guess about the level of abusive content in the question Note that a high Junk Detector score would have prevented the question from ever being displayed at all

Inputs related to content creation

When an author posted a question, the system increased the total number of ques-tions submitted by that author by 1 (Quesques-tionsAskedCount) This configuration allowed new contributors to start with a reputation score based on the average quality of all previous contributions to the site, by all authors (AuthorAverageQues tionQuality).

When other users answered the question, the question itself inherited the AverageAnswererQuality reputation score for all users who answered it (If a lot of good people answer your question, it must be a good question.)

Inputs contributing to positive reputation for an answer

As with a question, several events could indicate that an answer was valuable to the community When a reader took any of the following actions on an answer, the author’s AnswerQuality reputation score increased:

• The author of the original question selected the answer as Best Answer

• The community voted the answer Best Answer

• The average community rating given for the answer

Inputs contributing to negative reputation for an answer

If the number of negative ratings of an answer rose significantly higher than the number of positive ratings, the system hid the answer from display, except to users who asked to see all items regardless of rating The system lowered the AnswerQual ity reputation score of answers that fell below this display threshold This choked

Inputs.