The greatest of all time in the AFL - a quantitative approach

Using RAPM to evaluate AFL players 1897-2019

The debate about who is the greatest of all time in any given sport is an oft-debated topic. In some sports, the question is less controversial, for instance cricket, basketball, Formula One, billiards all have fairly unambiguous GOATs, but for others, such as Australian Football, the answer is much less clear. So who are the main contenders? It's difficult to even narrow it down to a handful. Tony Lockett stands alone at the top of the all-time goal-kicking leaderboard, but this is exclusively the domain of forwards. Ted Whitten was named captain of the the official AFL Team of the Century. AFL journo Mike Sheahan famously placed Wayne Carey ahead of all comers when he published his best players of the past 25 years. Hayden Bunton, Dick Reynolds, Ian Stewart and Bobby Skilton remain the only triple Brownlow medalists, although current Fremantle captain Nat Fyfe is a good chance to join them. There's also Leigh Matthews, a popular choice amongst footy fans, who managed to kick over 900 goals, despite playing a significant portion of his career in the midfield.

Like in many other sports, the lack of good ‘box-score’ statistics prior to the modern era makes settling the debate quantitatively, exceedingly difficult. In the AFL, box score data before the 1970s either wasn't recorded, or just isn't publicly available, and detailed player-level stats weren't really recorded until the late 1990s. Goals scored however, are recorded at the player level all the way back to 1897 and this has promoted the great full-forwards into the GOAT conversation a little more frequently than is perhaps warranted. To illustrate this bias: any fan with a cursory knowledge of the history of the game can probably name a handful of great full-forwards from an era that predates their birth, but these same fans will likely struggle to replicate the exercise with full-backs. Some good analytical work has been done by InsightLane comparing the great goalkickers across eras, after adjusting for various scoring trends. They find, very reasonably, that on an adjusted goals per game basis, John Coleman pips Peter Hudson as the greatest full-forward of all time. However the full-forward is one of 18 players on the ground, and I believe that the ease of measuring their contribution, does not necessarily mean they are more valuable than positions, whose contributions are less easily quantified.

The Approach

In an attempt to provide a quantitative answer to the debate, I’m going to use a method called RAPM, short for regularised adjusted plus minus. RAPM has been around for a while and has traditionally been applied to ice-hockey and basketball. You can read a detailed, technical description of the method here, or watch Ben Taylor's excellent NBA-specific explanation here for more information. Very simply, RAPM is a statistical method that attempts to untangle the complex interactions of individuals within a team and assign numerical values to each player that represent that players impact or value to their team. RAPM credits the player for playing in a winning team, particularly if the team did worse when the player was absent, and punishes the player for being part of a losing team, particularly if the team performed better when the player missed. In addition RAPM pushes players with short careers towards zero, and as such, will rate players more highly who remain in successful teams for long periods.

In basketball, this approach is rather effective, since:

1. Granular substitution data allows for a large sample of a teams performance with and without each player

2. There are only five players on the court at one time, meaning less noise in each observation

3. Each team plays over 80 games per year

However, even in the context of NBA, RAPM results can be unstable and yield some pretty counter-intuitive results, whereby players who switch teams at the right (wrong) time, get undue credit (blame) for a rise (fall) in their teams’ fortunes. There are various solutions to this, such as anchoring your ratings to a Bayesian prior, but I'll leave further discussion of this to a future blog post. Applying RAPM to AFL, where there are more players on the field, less games per year and no granular substitution data may not be a great idea and could yield very implausible player ratings. But to steal the pithy wit of Churchill: RAPM may be the worst approach to evaluating AFL player ability since 1897, except for all the rest! So here goes…

The Data

I parsed data for this analysis from AFL Tables, which is a great resource and contains an impressive collection of game statistics going right back to 1897, including importantly, complete team lists for each match. In order to allow for cross-era comparisons, a simple adjustment to the raw data must be made. As can be seen, scoring rates have varied over time. At the beginning of the 20th century, scoring was low and it gradually increased until it peaked in the 80s / 90s before declining.

Unaccounted for, this trend will inflate the ratings of good players who played during high-scoring periods and deflate the ratings of good players who played in low-scoring periods. Thus, I adjust the data, such that the average and standard deviation of points scored is fixed for each season in current terms (2019). Conveniently, this ensures the model will give each player a rating that can be loosely interpreted as their 2019 point value, that is, how much better they make their team, measured in current-day points. A couple of points about the analysis before discussing the results:

1. I choose to model it in terms of points scored / points conceded in order to give each player an offensive and defensive rating, the sum of which is their overall value. This is often done in basketball and produces richer, more interesting results, that can be sense-checked faster, e.g. do the great fullbacks provide value defensively, and the great full-forwards offensively?

2. I assigned each player a decade grouping. For players whose careers spanned more than one decade, they were assigned the decade in which they played the most games.

The Results

The model produced offensive and defensive ratings for 12,754 players. The range of ratings spans a 15 point range from roughly -7.5 to +7.5, although 99% of players fall in the [-3, +3.9] interval. A visual of the distribution can be seen below, with the rather vibrant colour indicating the decade. Satisfyingly, the distribution seems relatively smooth and symmetrical, with a perhaps as slight positive skew and a reasonable spread of players from each era. Note that the player ratings represent the average value of the player over their whole career, so players who retire closer to their peaks, make receive favourable ratings (sample size notwithstanding), than those who push on until they are dropped to the reserves.

Okay, enough of the boring stuff, so who is the GOAT?

So RAPM makes two-time Brownlow medallist Chris Judd the GOAT of Australian Football, with a value of 7.22 points above the average player. It's a bit of a surprising choice, but when the field is wide open, any one candidate winning is a surprise in some respects. In what will bring a tear to the eyes of all defenders, star full-backs Dustin Fletcher and Stephen Silvagni both feature in the top-five. Michael Tuck, the legendary Hawthorn captain who won seven premierships comes third. Then we get our first red flag. Brent Guerra? What? Really?? He did win two premierships with the Hawks, but I doubt even his mother would have him has high as fifth in the AFL GOAT list. He wasn't even the best defender in the Hawks team at the time! So clearly, some of these results are very noisy. Shane Mumford (18th) and Travis Varcoe (21st, just off the chart) both began their careers with Geelong during a period of rapid improvement and are likely given undue credit for the transformation, at the expense of more seasoned players and then moved onto teams which were also successful. Heath Shaw too, has been the blessed with good fortune in where he has played. In fact, one commonality between all the players in the top 20 (there are a couple I haven't heard of, who could be exceptions), is that they all played for great teams. In a couple of instances one could argue, that the player was the driving reason behind their teams' success, but in most, they were the beneficiary of having great teammates. How do the champion players mentioned in the opening paragraph fare?

• Wayne Carey: +4.27 (99th percentile)

• Leigh Matthews:+2.93 (98)

• Dick Reynolds: +2.21 (95)

• John Coleman: +2.21 (95)

• Peter Hudson: +2.20 (95)

• Bob Skilton: +2.13 (95)

• Tony Lockett: +1.92 (94)

• Nat Fyfe: +1.62 (91)

• Ted Whitten: +1.00 (85)

• Ian Stewart: +0.30 (69)

• Haydn Bunton: -0.20 (37)

Haydn Bunton's rating seems very unfair and is probably a consequence of him playing for Fitzroy during the 1930's - a time when the club played no finals matches and that he may have played his best footy in the WAFL, where he won three Sandover Medals. Ian Stewart also seems low considering his three Brownlows, but the rest seem to be in the right ballpark. I'm pretty sure I know what's going on here. If you only could see Carlton's results over the last few years, you wouldn't consider Patrick Cripps a great player. But that's how RAPM works, and that's why it can be used to rank and evalulate players going back to era when no statistics (apart from the score) were kept. To get really accurate ratings, you need additional information, such has box-score statistics, and that context, for players like Bunton, is missing. Perhaps I'll look at including some sort of Bayesian prior in a future post, that accounts for Brownlow votes, B&F wins, or box-score stats where available. But for now, we're stuck with the RAPM numbers. Here's a bubble plot, charting all the players with at least 100 games experience, with name labels for the outliers.

There's a lot of info packed into that graphic and suffice it to say, one could spend hours evaluating the credibility of each rating. There are clearly a lot of issues with the ratings (Adam Treloar stands out) - and they definitely shouldn't be relied upon as a complete player ratings systems - but I find it pleasing that it seems to have captured some kernels of truth, e.g. Robert Shirley a renowned tagger for Adelaide in the 2000s rated elite defensively but below average in attack. Zac Dawson, favourite defensive player of Ross Lyon also ranks elite defensively, but below average offensively. Star full-backs Dustin Fletcher and Stephen Silvagni are near the top of the league defensively. Gordon Coventry rates average defensively, but is amongst the highest rated offensive players of all time.

For those wishing to go back through the decades, I've added a chart below with the top ten players in each era.

Overall, I'm pleased with the results of the experiment. The ratings seems to peg the vast majority of players in the right ballpark, it's just that the few that are wrong really stand out. I knew that RAPM has issues, and that these issues would be exacerbated by the nature of AFL data, so I expected some wacky results. And for most if not all of the red flags I came across, I can understand why the model gets them wrong. Going to leave it there.