2008-02-29, Robert Jasiek

Quality of SOS

Preface

The tiebreaker SOS is defined as the Sum of Opponents' Scores.
Before using a tiebreaker, one should make the more fundamental decision whether to either share final places or break ties. If ties shall be broken, then one hasto choose a tiebreaker.
A tiebreaker is used within a particular tournament system and its aims. Therefore every tiebreaker has to be reevaluated for its specific usage within everyparticular tournament system and aims.
Tiebreakers can be used for different purposes: seeding, pairing, final results ordering.
An ordered combination of successive tiebreakers like SOS - SOSOS might be considered to form a new tiebreaker and should be analysed afresh.
For the sake of simplicity, in the following it is generally assumed that ties shall be broken, SOS is chosen as the tiebreaker, the behaviour of SOS is similar for allparticular tournament systems and its aims if only it can be applied there at all, SOS is considered mainly for the final results ordering, and SOS is used as the first and only tiebreaker. Future research should study also other assumptions in greater detail.

Advantages of SOS Itself

Relation of Greater SOS And Greater Probability of Strength

When player A has a greater SOS than player B, then on average the probability that A has played against "stronger" opponents on average than B is greater than 0.5. The greater the SOS difference between the two players is the greater the probability is on average.

In practice, it suffices to understand intuitively what the meaning of "stronger" might be. Greater strength of opposition also implies greater strength of the player (for a constant score). Precise values of such probabilities are not known so far. However, if the SOS difference is considerable, then the probability could be estimated as being significantly greater than 0.5.

SOS is in Line with the Primary Scoring System

For a single player, greater SOS for him than smaller SOS for him can be interpreted as greater strength of his opponents during the tournament. If a scoring system finds the "better" player, then SOS finds the "tougher" opposition. SOS measures the same thing in opponents that the score measures in players.

Advantages of the Social Context around SOS

SOS is Popular

Every pairing program offers SOS. Some set it as default tiebreaker.
Some associations or federations have it among their recommended tiebreakers.
Many tournaments use it.

Disadvantages of SOS Itself

Earlier Wins are Rewarded More than Later Wins

It is unclear why winning in earlier rounds should be more important than winning in later rounds (with early wins a player collects more SOS since he gets to play players with more wins earlier).
A priori one wants to consider all rounds equally important - the behaviour of SOS opposes this.
Since in a Swiss type tournament players with equal numbers of wins are paired against each other, competitors for the top final places are paired against each other the more likely the later the round. So the opposition becomes more and more worthy in later rounds while SOS decreases rewarding of wins in later rounds.

Greater Numerical Precision than Significance

Almost always players with similar SOS have such a small SOS difference that it is smaller than every meaningful numerical significance would be. In other words, SOS pretends to be more accurate than it is.

The exact significance of SOS is unknown (and also depends on the tournament system, round, etc.) but a small SOS point difference is essentially meaningless. It is doubtful to decide winners on small SOS differences.

A Player Cannot Influence His Opponents' Later SOS Changes

It is unclear why it should be fair that one's opponents may get as many or few wins in rounds after one has played them (so that one cannot influence their later achievements during the tournament). For mathematical details, see the later research chapter.

Dependency on Pairing Luck

During the early rounds of a tournament, SOS does not provide or not provide sufficient information for a pairing program to let it decide well about the pairings. In return, how great a player's SOS is in early rounds also heavily depends on so called pairing luck.

Mirror Effect

Besides, at the very top or bottom of the tournament table, the boundaries of the player field lead to "mirror" errors: Players at the very top / bottom cannot always get opponents with the same performance of the first placement criterion because such (new) opponents might not exist.

Pairing Strategy Can Contradict SOS

It is unclear why SOS should distinguish players at the end of a tournament after the pairing program has done its very best to make SOS of every two players with the same number of wins as close as possible. While the relevant pairing strategy is generally considered good, its side-effect on SOS in the final tournament result list is counter-productive.

Doubtful Comparison of Every Two Players

As a consequence of the disadvantages "Earlier Wins are Rewarded More than Later Wins", "Greater Numerical Precision than Significance", "Pairing Strategy Can Contradict SOS", comparing the SOS of any two players is doubtful.

Unclarity of Greater Importance of Won versus Lost Games

It is unclear whether winning or whether losing games against opponents with greater numbers of wins during the tournament is better. SOS is indirectly affected by this aspect because SOS rewards winning in early rounds more than losing in early rounds; a player gets the more opponents with previously the more won games the more games the player has won himself in the early rounds.

SOS is Not Fair on Average over Many Tournaments

Contrary to a widely spread myth - due to the Law of Great Numbers, SOS cannot be fair on average over many tournaments: It requires an infinite number of tournaments to allow that conclusion while no player ever can play an infinite number of tournaments. Even worse, specific titles are issued only once per year, tournament conditions and a player's development change.

SOS Does Not Break Ties in Some Tournament Systems

E.g., SOS does not break ties in a round-robin. (For reference, nor does SOSOS.)

Disadvantages of the Social Context around SOS

Manual Calculation is Tedious

Calculating SOS by hand correctly for all participants is tedious because one has to add a lot of numbers and check many references.

Commentary on the Quality of SOS

Although not every (dis)advantage carries the same relevance, it cannot be overlooked that SOS has more disadvantages than advantages and that most of the disadvantages are severe. SOS is popular nevertheless not just because of its established history but also because availability of alternative tiebreakers, which are not discussed here, is restricted. Most tiebreakers are even much more doubtful than SOS. The few interesting other tiebreakers have their merits but have a similar quality to SOS, i.e., are not so overwhelmingly better that everybody would swiftly prefer them.

When using SOS, one should be aware of its imprecision. Within one large score group, the player with the greatest SOS may often have performed better than the player with the smallest SOS. However, the exact order of all players in that score group should not be trusted because the typically small SOS differences between two adjacent players in the final results table are often smaller than the significance of SOS can be.

Research on One Significance Aspect

Here only one of the significance aspects is studied: "A Player Cannot Influence His Opponents' Later SOS Changes". The other significance errors would have to be added.

Maximal Imprecision

Given a Swiss tournament with N (even) players Pn; n=1..N, R rounds, all players playing all rounds, no pair occurring twice, opponents Onr; r=1..R of player Pn in round r, no Byes, scores Snr; r=1..R in {1;0} representing {win, loss} of player Pn in round r, Sn the player's score vector for all rounds, a simplifying assumption that a player with sum[i=1..r-1]Sni points before the start of round r gets in round r an opponent with also sum[i=1..r-1]Sni points, a simplifiying assumption that all Onr get an average of 0.5 points in rounds i>r.

Then after R rounds player Pn has the expected SOS

sum[for_opponents_in_all_rounds] (opponent's points before actual round + opponent's points in actual round + opponent's points in later rounds) =

sum[r=1..R] ( sum[i=1..r-1]Sni + 1-Snr + 0.5*(R-r) ).

The latter summand could be 0*(R-r) if all Onr would lose in all later rounds or 1*(R-r) if all Onr would win in all later rounds. Therefore the maximal imprecision of the expected SOS of Pn is +- sum[r=1..R] 0.5*(R-r).

Remarks

The minimal imprecision of SOS is 0. One approach to assessing some average imprecision would be consideration of a reasonable model like the binomial distribution where the minimal imprecision 0 has the greatest frequency and the positive or negative maximal imprecisions each have the frequency 1. Doing this is subject to further research.

The maximal imprecision is independent of Sn!

Closest SOS differ by 1 (if we ignore jigos). Contrarily, this imprecision (error) of SOS can become very great in comparison! So one can hardly assign any reasonable grade of significance for SOS. E.g. in a tournament with R=5, why should a SOS difference 4 between two players be significant but not a SOS difference 3? However, the most clearly the often used significance of a SOS difference 1 (even disregarding R(!)) cannot be justified.

Winning earlier is advantageous since the first summand is greater then.

To conclude, it is hard to justify usage of SOS as the first tiebreaker in a Swiss (or a MacMahon) tournament. SOS may depend on opponents' performance during a tournament, however, the dependency is by far too weak for meaningful numbers. Paul Matthews has shown that it is better than drawing a lot but this is about the best that could be said about SOS. It is much more honest not to use any tiebreaker than to use a particular tiebreaker with such an extremely high error per [used] significance ratio.

Example

R=5 and Sn=(1,0,1,0,1).

The expected SOS of player Pn after round 5 is
2+3.5+2+3.5+2 = 13.
Its maximal imprecision for this player is
+- (2+1.5+1+0.5+0) = +- 5.
So after the tournament Pn might have a SOS from 8 to 18.