ESPN's Player Rater: How is it calculated? Why does it matter?
by Mac Squibb
February 21, 2019
Anyone who has played fantasy baseball on ESPN has likely seen what the site calls the Player Rater. For those who haven’t, it’s a collection of values assigned to each player, based on the scoring categories of the league, which are totated and used to rank the players.
But is the Player Rater a valuable tool for fantasy?
How is the Player Rater Calculated?
To determine if the Player Rater is a valuable tool, it’s important to first understand how the it’s calculated. Through trial and error and some research I have attempted to replicate the values that the Player Rater produces. The results show that each category of the Player Rater is calculated based on a statistical measurement known as a z-score.
I’m not going to go into the math involved, you can to read about that here, but essentially a z-score shows how much better or worse a value is relative to all other values. All counting stats such as runs, home runs, RBIs, stolen bases, wins, strikeouts, and saves are calculated this way. Ratio stats are still based on z-scores, however, they also include a playing time component that weights the results. It should make sense that a .300 average in 100 at-bats is less valuable than a .300 average in 500 at-bats, just as a 3.50 ERA in 10 innings is less valuable than a 3.50 ERA in 100 innings. This principle applies to all ratio stats used on ESPN including AVG, ERA, and WHIP. Despite my many efforts, I have yet to figure out exactly how the playing time component is incorporated into the calculation of the z-scores. I even tried calling ESPN, but it turns out that the calculations are proprietary information. So, in order to model the Player Rater for ratio stats I decided to apply a machine learning technique called k-Nearest Neighbor (kNN). An example of the kNN algorithm: Pretend that we would like to predict Player X’s Player Rater value for batting average. The algorithm would function as follows:
(1) Get Player X’s projected at-bats and batting average
(2) Find the players whose at-bats and batting averages were most similar to Player X’s projections
(3) Take those players’ Player Rater values and average them
The resulting value is what the algorithm would predict to be Player X’s Player Rater value for batting average. The algorithm would function the same with ERA and WHIP, but would use innings pitched instead of at-bats.
Why does the Player Rater matter?
Now that we know how the Player Rater is calculated, it’s important to show why the Player Rater is a valuable fantasy baseball tool. Using the Player Rater as a way to evaluate players has several benefits, first of which is that it allows us to compare the overall production of different players. Have you ever wondered just how valuable a player with 40 steals is compared to someone with 40 home runs? Not a problem with the Player Rater. What about a player with a high batting average and 10 stolen bases compared to someone with a low batting average but 30 stolen bases? Again, not a problem. It even works as a way to compare hitters and pitchers despite accumulating stats in different categories. The reason that we can make these comparisons with the Player Rater is based on what a z-score represents. The goal of fantasy baseball is to accrue the most and best statistics which, by definition, will also produce the highest z-scores. Therefore, the higher the Player Rater value, the more valuable that player is in fantasy.
The largest benefit of using the Player Rater may be its ability to cut through any biases that we, or others, may have. Most rankings, including those displayed on draft softwares, are created subjectively and thus exposed to the biases of the person creating them. We also have our own personal biases that can easily cloud our judgment when evaluating players. The Player Rater, however, is an objective measure of a players performance that should allow fantasy owners to see past biases.
The Player Rater itself has several limitations that should be noted as does the kNN algorithm. A common complaint I’ve seen and heard is that the Player Rater appears to overvalue steals. While this isn’t true as the Player Rater calculates every value the same, the high variance in stolen base totals does create a situation that individuals need to be aware of when using the Player Rater. It should be obvious that there comes a point when accumulating more of a certain stat provides increasingly less value. An example of this would be drafting speedsters Dee Gordon, Billy Hamilton, and Mallex Smith all to the same team. While their Player Rater values indicate that they have relatively high value, all of them are propped up by their stolen bases totals. Thus, having all three on the same team is overkill and actually diminishes their overall value. This is a concept that I think comes naturally to most of us, but is still worth noting.
The kNN algorithm experiences limitations due to the fact that I currently only have data from the 2018 season. The issues with the algorithm only manifest themselves at the extreme ends of the spectrum and can be seen most seen more glaringly with Corey Kluber.
Kluber is projected by the kNN algorithm to have the best Player Rater ERA and WHIP values despite being projected for a 3.46 ERA and 1.12 WHIP by Depth Charts. The reason for the discrepancy is that there were so few pitchers who threw 200 or more innings in 2018 for the algorithm to compare him to. Last year, there were just 13 pitchers who threw 200 or more innings. Here is a list of all of their ERA’s and WHIP’s along with Klubers projection for 2019.
As you can see, of the players most similar to Kluber, in terms of innings pitched, almost all of them had significantly lower ERA's and WHIP's. Kluber’s projection falls into a gap in the data which caused the kNN algorithm to assign a higher value to Kluber than he deserves. As more data is collected these gaps with be filled in and the kNN algorithm will become increasingly more accurate.
ESPN’s Player Rater is a tool that fantasy baseball players should use with confidence going forward. The system allows us to compare different types of players and also helps us remove biases from our evaluations. While it does have limitations, if aware of them, the system can be an incredibly valuable tool during draft preparation and throughout the season.
I have taken the Depth Charts projections at Fangraphs and calculated each players’ projected Player Rater value for 2019 which you can view here. In the near future, I plan to do the same with the ATC and THE BAT projections. I also hope to include an average of the three systems which should help mitigate the limitations of the kNN algorithm.
Check out other articles