Understanding Majority Judgement

Title: Understanding Majority Judgement

This article presents majority judgement, a voting method based on choice evaluation and median result. We also explain how to use this method directly within Slack conversations with our Open Agora for Slack polling application.

Why Use Scores?

We have already described in a previous article how ranking choices of a poll was more powerful than approbation voting. Whether Condorcet method or variants like Instant-runoff are used, a better consensus is extracted from the set of votes. This brings a better representation of a group opinion, thus a more accurate and accepted decision.

Condorcet -Serment du jeu de Paume

Further reading

To be even more expressive, voters can quantify their rankings. Open Agora for Slack latest update introduces evaluation polls, allowing to associate “scores” with choices.

Evaluation poll sample in Slack
Evaluation poll example: voters should quantify how much they appreciate each choice

But as meaningful as they may be, detailed and expressive votes mean nothing if the result is not computed accurately. Indeed, in general, such scores are assembled in a basic way, using the sum or average of all scores.

In that case, the result is very sensitive to strategic voting. Strategy can be under-evaluating a choice to be almost certain it is not selected, or over-evaluating another to maximize its chances. These non-representative scores are counted in the final sum or average score of the choice, leading to a biased result.

Majority Judgement

To avoid that bias, Balinski and Laraki proposed in 2007 another way to compute the result[1], called majority judgement. The basic idea is to take the median of all scores received by a choice (i.e., the highest score selected by half of the voters). Strategic voting consequences are reduced by the fact that extreme values are counted as 1 vote, the effective score not being used.

Evaluation poll result with majority judgement in Slack
Result legend
Evaluation result: summary of votes, and computation of the median score for each choice

In our Slack app, this median result used by majority judgement is shown along the 50% vertical line of the bar chart above. It represents the minimum evaluation of half of the voters.

  • The number of votes at each score is displayed in a horizontal block, for each choice, using a different color.
    See legend for more information on what score it represents.
  • If median falls between 2 blocks (like for choice 2), the result is the average value of the 2 blocks.

As you may notice, if the final median score of a choice is 3 (choices 1 and 4), the respective number of and 1’s and 2’s (or 4’s and 5’s) it received has no influence on the final result. As a consequence, strategic voting becomes inefficient.

In addition, evaluation polls enhance ranked polls since they enrich the result with a clear, quantified satisfaction rate. While Condorcet extracts a (good) consensus from the different voters’ rankings, it does not really tell how much voters like the winning choice. Indeed, ranking a choice as first option only means it is the preferred one. In an evaluation poll, it could have been expressed with a 5 (best choice ever!) or a 3 (ok, why not). Their result can then be more useful for managers to decide what to do, and whether this poll is meaningful.

Scoring Ranges

In its initial definition[1], scores (named “grades”, or sometimes “ratings”) of majority judgement range from “bad” to “excellent”, in 7 steps. This was discussed later in other articles (for example in [2]).

  • Why 7 steps, why no more or no less?
  • Why an odd number of steps (which always contains an intermediate, neutral value)?

We chose to let our Slack app users decide, by allowing different scoring ranges for majority judgement. Ranges can be selected at poll creation, according to the poller use cases:

  • 👎 / 👍 : binary choice, like or reject choices, similarly to approbation voting
  • 😢 / ☹️ / 🙂 / 😃 or ⛈️ / 🌧️ / ⛅ / ☀️: extend the previous mode with 2 extreme values, for a more qualified opinion expression
  • ⭐ to ⭐⭐⭐⭐⭐, 0 to 5 or 1 to 10 : quantify how much you appreciate choices

Note that in the first 3 options (thumbs, smileys and weather), there is no neutral value, to force users to like or dislike choices. Dislikes are expressed with negative scores: for a better visual representation, these are shown in orange on our result charts, positive scores in green.

In the last 3 options, scores map to their integer value, which may possibly include 0 as lower bound.

Specific Adjustments

Similarly, we had to make specific adjustments in our app in two specific cases.

Partial Votes

What should we do if for some reason, a voter does not give a score to some choices? The common usage is to select the worst-possible score for this choice automatically[1,2] (meaning “bad/reject”). However, this is not always suitable in our case, since Slack UX does not allow easily to force a voter to evaluate all choices at once. And even if it was, would it be adapted to all use-cases?

We think that most of the times in non-political situations, abstention does not mean “reject” (which is easy to express with a vote, anyway), but rather “ignore/don’t know” (which is different from “neutral”). So, when we compute the median, we only consider expressed votes, even if all choices have not received the same number of votes.

This does not appear on the result charts, which represent votes as percentages of the total amount of votes for a choice. However, users should be aware that the underlying amount of votes may not always be the same.

Tie-breaking

Using the median to rank choices often produces ties. Indeed, for 4 available scoring options, the median can take only 7 different values.

Breaking these ties becomes a necessity, and can be done in several ways.

Option 1[1]: for all tied choices, remove all votes equal to the median score one by one, recompute the median at each step until there is a winner.

Option 2[1] (computationally-efficient approximation of option 1): for all tied choices, compute the number of votes below (A) and above (B) the median, take the highest percentage:

  • if it is a “lower” percentage A, this choice loses over the other ones;
  • if it is a “higher” percentage B, this choice wins.

The chart below illustrates this method on choices 1 and 4:

Tie-breaking (option 2) majority judgement result
Breaking tie with option 2: computing extreme percentages

The highest percentage is A1 = 40%: choice 1 loses over choice 4, since there is a higher proportion of voters disliking the choice than anything else.

Option 3suggested in [1], formalized in [2]: count the proportion of voters which have an evaluation at least equal to the median.

In our example, this percentage is 70% for choice 4, 60% for choice 1: choice 4 wins with a higher satisfaction rate.

Our implementation of majority judgement inside Slack uses option 3. Indeed, it seems to us that it favors global satisfaction (hence consensus) over extreme ratings (what options 1 and 2 tend to emphasize). In the previous case, the final result is the same whichever tie-breaking option is applied. There are some more unconventional cases where they may be different, and where we prefer to highlight consensual choices.

Conclusion

We believe our app brings the benefits of majority judgement to everyone using Slack, without adding complexity in the creation, voting or result process: try it, use it, and let us know what you think about it!

Note that at any time, other result algorithms (sum, average, Condorcet) can be displayed within the application to help you get a better view on your workspace opinions.

References

[1] M. Balinski and R. Laraki (2010). Majority Judgment – Measuring, Ranking, and Electing. MIT.
Initially printed in: M. Balinski and R. Laraki (2007). A theory of measuring, electing and ranking. Proceedings of the National Academy of Sciences USA, vol. 104, no. 21, pp. 8720-8725.

[2] S. Bosworth, A. Corr and S. Leonard (2019). Legislatures Elected by Evaluative Proportional Representation (EPR). Journal of Political Risk, vol. 7, no. 8.

Partager / Share this...