An exciting addition to our Challenge
Here is how it works. f you have been receiving the Oracle Rankings Newsletter or reading the site, you are used to seeing how many games we got it right and wrong in this format:
[51%]: (12) Seattle Seahawks 21 @ (13) Dallas Cowboys 12
[67%]: (4) Pittsburgh Steelers 34 @ (29) Houston Texans 6
[84%]: (21) Oakland Raiders 10 @ (1) Philadelphia Eagles 19
Here, we missed our pick for DAL but were correct on picking PIT and PHI to win. Thus, the Oracle was 2-1 or picked 66% correctly. This is the accuracy (#right/#total picks) and this is how we keep score for the confidence section of the challenge.
One can argue that if we picked DAL to win at 51%, we were not very confident and being incorrect was not that big of a deal. Meanwhile, we picked correctly PIT and PHI, but we were more confident on the PHI pick, so we deserve to get more credit for that. This leads to the problem to measure the success of probabilistic predictions. The solution is to introduce the Quadratic Score (QS) to the Trinity vs Oracle competition. It assigns a value to the confidence of your prediction and it has the advantage to be a proper scoring function.
The idea is that one will gain points if one picks correctly and has high confidence while one will lose points if one picks incorrectly. You will not lose any points if you make a pick and have a 50% confidence since in that case, you were not sure.
Some Math: Let us denote ProbWin = probability one assigned to the winner of the game. In our example, ProbWin(SEA@DAL) = 49%, since by picking DAL to win at 51%, we give SEA 49% chance of winning. Also, in the last game, ProbWin(OAK@PHI) = 84%.
We define the Quadratic Score (QS) to represent the quality of the pick as:
Note that QS(0.5) = 0 and multiplication by 10 above is just to improve readability of the results.
Let us now consider two prediction methods and compare their accuracy and quality of predictions
Method A:
[55%]: (12) Seattle Seahawks 21 @ (13) Dallas Cowboys 12 [ProbWin = 45%]
[60%]: (4) Pittsburgh Steelers 34 @ (29) Houston Texans 6 [ProbWin = 60%]
[64%]: (21) Oakland Raiders 10 @ (1) Philadelphia Eagles 19 [ProbWin = 64%]
Method B:
[75%]: (12) Seattle Seahawks 21 @ (13) Dallas Cowboys 12 [ProbWin = 25%]
[72%]: (4) Pittsburgh Steelers 34 @ (29) Houston Texans 6 [ProbWin = 72%]
[80%]: (21) Oakland Raiders 10 @ (1) Philadelphia Eagles 19 [ProbWin = 80%]
For the three games, both methods have the same accuracy, 2-1. The question of quality can now be addressed as follows:
QS(A) = QS(.45) + QS(.60) + QS(.64) = 6.3
QS(B) = QS(.25) + QS(.72) + QS(.80) = 2.7
Since QS(A) > QS(B), and we say that Method A is a better prediction method than Method B
The Oracle now challenges you to not only be more accurate but also to have a better quality of picks this season. So, if you decide to add confidence to your picks, you will have a scale to decide the probability for the winner
