Saturday, 11 February 2017

The Wisdom of Crowds: A Census of EPL Forecasts

Introduction


We're nearly two-thirds of the way through the 2016/17 EPL season, which seems a good time to try to predict what might happen. Chelsea’s nine-point cushion and relentless form make them clear favorites for the title; not since Newcastle in 1996 have a team blown such a lead. Just five points separate second from sixth as the remaining superpowers battle for Champions League places: who will miss out? Perhaps the mantra ‘most competitive EPL season ever’ is best reserved for the relegation fight, though. Six teams, two points and an ever-changing landscape. Amongst them: last season’s heroes, Leicester. Too good to go down?

Most TV pundits are definitive in their predictions, indeed they are typically paid to be so. Others prefer to let the numbers do the talking. Football analysts around the world build mathematical models to measure team strength and calculate the probability of match outcomes. Rather than saying “team A are likely to beat team B”, they'll say “I estimate that there is an 85% probability that team A will win”.

There is no agreed method for designing a forecast model for football. Consequently, predictions vary from one model to another. However, there is also strength in diversity. Rather than comparing and contrasting predictions, we can also collect and combine them to form a consensus opinion.

Last January, Constantinos Chappas did just that. Following gameweek 20, he collected 15 sets of predictions, averaging them to produce a ‘consensus forecast’ for the outcome of the 2015/16 EPL season. His article was published on StatsBomb here; we’ll return to the success of last year’s predictions at the end. First, I’m going to repeat the exercise for the 2016/17 EPL season. What do the combined predictions say this time around?

Participants


In total there were 15 participants this year, many of whom offered up their predictions in response to my twitter appeal. A big thank you goes out to (in no particular order):

@8Yards8Feet, @petermckeever, @goalprojection, @11tegen11, @cchappas, @SteMc74, @fussbALEXperte, @SquawkaGaming, @EuroClubIndex@opisthokonta and Sky Sports (via @harrydcarr)

To these, I added forecasts from the FT and FiveThirtyEight; I haven’t been in contact with them personally, but their forecasts are publicly available. I also added a bookmaker’s average, calculated by collecting the odds published on oddschecker.com and averaging the implied probabilities. That’s 14 - the final participant was myself (@eightyfivepoint).

The Predictions


Before we get into the results, a little bit about how they’ll be presented. I’ve followed last year’s article and presented forecasts as box-plots. These are a simple graphical representation of the distribution of forecasts for a particular outcome. The height of the shaded area represents the interquartile range: the 25th to 75th percentiles. By definition, half the forecasts lie within this range -- it provides a decent estimate of the variablity of the predictions.  The black horizontal line in the middle is the median (50th percentile), I’ll sometimes refer to this as the consensus forecast. The ‘whiskers’ extending out vertically from each box show the 5th to 95th percentiles. All but the highest and lowest forecasts for a given outcome will lie within this range.

On each plot I've also plotted the individual predictions as coloured points. They are identified by the legend on the right.

So, without further ado, here are the forecasts for this 16/17 EPL season.

The Champions



Not surprisingly, Chelsea are the clear favourites: the median forecast gives them an 88% chance of winning the league, as do the bookmakers. There’s not a huge amount of variability either, with the forecasts ranging from 80% to 93%. If Chelsea do suffer some kind of meltdown then it’s probably Spurs or City that would catch them, with median predictions of 5% and 4%, respectively. Liverpool and Arsenal are rank outsiders and any of the other teams finishing top would be an enormous surprise.

The Top Four



Now this is where things get a bit more interesting. Chelsea seem almost guaranteed to finish in the Champions League places, which leaves five teams fighting it out for the remaining three. Tottenham and Man City are heavily favoured: both have a median probability of at least 80% and the whiskers on their box-plots do not overlap with those of the next team, Liverpool.

The real fight is between Klopp and Wenger. Statistically they are almost neck-and-neck, with their box-plots indicating that the individual predictions are broadly distributed. Look closely and you see an interesting negative correlation between them: those that are above average for Liverpool tend to be below average for Arsenal (and vice-versa). You can see this more clearly in the scatter plot below. The reason must be methodological; to understand it we’d have to delve into how the individual models assess the teams' relative strength. Note that the bookies are sitting on the fence - they've assigned both Arsenal and Liverpool a 53% chance of finishing in the top four.


Man United are outsiders, but the consensus forecast still gives them about a 1 in 3 chance of sneaking in. Interestingly, the bookmakers odds – which imply a 44% chance of United finishing the Champions League positions - are way above the other predictions. Perhaps their odds are being moved by heavy betting?

The Relegation Candidates



Two weeks ago it looked like Sunderland and Hull were very likely to go down. Since then, the relegation battle has been blown wide open. The first six teams seem set for a nervous run-in and neither Bournemouth nor Burnley will feel safe.

The principal candidates for the drop are Sunderland, Hull and Palace, all of whom have a median prediction greater than a 50% chance of relegation. There is clearly a lot of variability in the predictions though, with the Eagles in particular ranging from a 38%-74%. You can certainly envisage any one of them managing to escape.

The next three clubs - Middlesbrough, Swansea and Leicester - are all currently level on 21 points, yet the median predictions imply that Middlesbrough (42%) are nearly twice as likely to go down as Leicester (22%). I suspect that this is because some models are still being influenced by last season’s results (for instance, Leicester's forecasts appear to bunch around either 15% or 30%). The amount of weight, or importance, placed on recent results by each model is likely to be a key driver of variation between the predictions.

What about <insert team’s name here>?


The grid below shows the average probability of every EPL team finishing in each league position. Note that some of the models (such as FiveThirtyEight, Sky Sports and the bookmakers) are excluded from the plot as I wasn’t able to obtain a full probability grid for them. Blank places indicate that the probability of the team finishing in that position is significantly below 1%.

An obvious feature is that Everton seem likely to finish in 7th place. The distribution gets very broad for the mid-table teams: Southampton could conceivably finish anywhere between 7th and 18th.


Last year’s predictions.


So how did last years’ predictions pan out? Leicester won the league, but the median forecast predicted only a 4% chance of this happening (compared, for example, to a 40% chance that they would finish outside the Champion's League places). However, the top four teams were correctly predicted, with a high probability of finishing there having been assigned to each of Leicester, Arsenal, City and Spurs.

Down at the bottom, both Newcastle and Villa were strongly expected to go down and they did. Sunderland were predicted to have only a 15% chance of staying up, yet the Black Cats escaped again. Instead, Norwich went down in their place having been 91% to stay up. Other surprises were Southampton (7 places higher than expected), Swansea (5 higher) and Crystal Palace (down 7).

How good were last year’s forecasts, overall? This is a tricky question and requires a technical answer. The specific question we should ask is: how likely was the final outcome (the league table) given the predictions that were made? If it was improbable, you could argue that it happened to be just that – an outlier. However, it could also be evidence that the predictions, and the models underlying them, were not particularly consistent with the final table.

We can attempt to answer this question using last season’s prediction grid to calculate something called the log-likelihood function: the sum of the logarithms of the probabilities of each team finishing in their final position. The result you obtain is quite low: simulations indicate that only about 10% of the various outcomes (final rankings) allowed by the predictions would have a lower likelihood. It is certainly not low enough to say that they were bad, it just implies that the final league table was somewhat unlikely given the forecasts. A similar result this time round would provide more evidence that something is missing from the predictions (or perhaps that they are too precise).

A final caveat..


Having said that – models are only aware of what you tell them. There are plenty of events – injuries, suspensions, and managerial changes – of which they are blissfully unaware but could play a decisive role in determining the outcome of the season. Identifying what information is relevant – and what is just noise – is probably the biggest challenge in making such predictions.

I will continue to collect, compare, combine and publicize forecasts as the season progresses: follow me on twitter (@eightyfivepoint) if you'd like to see how they evolve.


(This is a piece that I wrote for StatsBomb; I've copied it here.)



Wednesday, 18 January 2017

Poor FA Cup crowds erode home advantage

I was struck by the poor attendances at some of the FA Cup 3rd round matches this month. 17,632 turned up to watch Sunderland vs Burnley, less than half Sunderland’s average home gate this season. It was a similar story at Cardiff vs Fulham, Norwich vs Southampton and Hull City vs Swansea, all of which saw crowds below 50% of their league average this season.

An interesting statistic was recently posted on Twitter by Omar Chaudhuri, of 21st Club (@OmarChaudhuri). If you take all 181 FA Cup ties that involved two EPL teams (ignoring replays and matches at neutral venue) since the 2000/01 season, you find that the home team won 46% of the matches and the away team 30%. However, if you look at the equivalent league match between the teams in the same season, you find that the home team won 52% of the matches and the away team 22%. Although the sample size is small, the implication is that home advantage is less important in cup matches.

Lower FA Cup crowds and diminished home advantage - are the two connected? This seems a reasonable hypothesis, but I’ve never seen it demonstrated explicitly. I aim to do so in this post.

Cup Matches vs League Matches


To answer the question I’ll look specifically at cup ties that involved teams from the same division, from League 2 to the EPL, and compare the outcomes to the equivalent matches in the league. This approach isolates the influence of any changes in circumstance between the two games – including lower or higher attendance.

I identified every FA Cup tie, from the third round onwards, that involved two teams from the same-division since 2000/01[1], along with the corresponding league match.  I then removed all matches at a neutral venue[2]. This left me with a sample of 357 cup matches, and the same number in the league.

I then measured what I’ll refer to as the home team’s attendance ratio -- their average home-tie FA cup attendance divided by their average home league attendance -- in each of the last 16 seasons. Season-averaged attendance statistics for both league and FA cup games (3rd round onwards) for every team were taken from www.worldfootball.net. Ideally, you would directly compare the attendance of each FA Cup tie with that of the equivalent league game. However, I don’t have the data for individual games, so instead I used each team’s season averages for cup and league as a proxy (but if anyone has this data and is willing to share it, please let me know!)

I used the attendance ratio to divide my sample of matches into three sub-samples: well-attended matches, mediocre attendance and poorly-attended matches. The former are defined as cup matches in which the crowd size was greater than 90% of the home team’s league average. A mediocre attendance is defined as a crowd size less than 90% but greater than 70% of their league average, and a poorly-attended one as less than 70% their league average. For each group, we’ll look at differences in the fraction of home wins, away wins and draws between the FA Cup ties and league matches.

Table 1 summarizes the results. Let’s look at the first three lines - these give outcomes for cup ties in which the attendance was at least 90% of the league average. There have been 148 such matches in the last 16 seasons: the home team won 56%, the away team 23% and 21% were draws. In the corresponding league matches, the home team won 51%, the away team 24%, and it was a draw in 26%. So, there was a small increase in the proportion of home wins relative to the league outcomes, with correspondingly fewer draws. In about a third of these ties the attendance was greater than their league average: the home side may have benefited from a more vociferous support.

Table 1

The next set of lines in Table 1 show the results for the FA Cup matches that had a mediocre attendance – those in which the attendance ratio was between 70% and 90% of the home side league average. The home team won 44% of these matches, which is slightly below the home win rate in the corresponding league matches. There is again a fall in the number of draws, but this time the away team benefits, winning 6% more often than in the league matches. The differences are small, but there is some evidence that the away team were benefitting from the below-average attendance.

However, the increase in away wins becomes much more striking when we look at poorly-attended cup matches: those in which the attendance was less than 70% of the home team's league average. The home team won only 34% of these ties, 14% below the corresponding league fixtures. The away win percentage increases to 42% and is 19% above the league outcome. Indeed, the away team has won poorly-attended cup matches more frequently than the home team. This is despite the home team winning roughly twice as often as the away team in the corresponding league fixtures (48% to 23%). The implication is very clear: when the fans don’t show up for an FA Cup tie, the team is more likely to lose. I don’t think I’ve seen any direct evidence for this before[3].

In all three sub-samples, it's worth noting that draws are down 5% relative to the corresponding league outcomes (although the beneficiary depends on the attendance). Presumably this is down to the nature of a cup tie: teams are willing to risk pushing for a win in order to avoid having to play a troublesome replay (or a penalty shoot-out during a replay).

So why are some fans not showing up? One obvious explanation is that they are simply unwilling to shell out more money beyond the cost of a season ticket. Maybe clubs should lower their prices for FA Cup matches; I’d be curious to know if any do. There could even be an element of self-fulfilling prophecy: the fans believe that their team have no real chance of winning the cup and so choose not to attend, to the detriment of their team. Perhaps the fans are aware that the cup is simply not a priority – their club may be involved in a relegation battle, for example – and that they are likely to field a weakened team.

The bottom line seems clear enough, though: if clubs want to improve their chances of progressing in the FA Cup they should ensure that they fill their stadium.


--------------------
Thanks to David Shaw, Jim Ebdon and Omar Chaudhuri for comments.

[1] Data was only available for all-Championship ties from 02/03, 08/09 for L1 and 09/10 for L2.
[2] Replays were retained, although the outcome of penalty kicks was ignored (i.e., a draw at the end of extra-time was scored as a draw). There are 64 replays in the sample in total, of which 8 went to penalties.
[3] One caveat is that the sample size is pretty small: this analysis could do with being repeated on a larger sample of games (and with the specific match attendances, rather than season averages). However, the increase in the away percentage in the smallest sample (attendance ratio < 0.7) is still highly significant. 

Tuesday, 10 January 2017

The Frequency of Winning Streaks

Thirteen – an unlucky number for some. So it proved for Chelsea: just one win shy of equaling Arsenal’s record, their thirteen-match winning streak was finally ended by an in-form Spurs side. While there may be some temporary disappointment amongst Chelsea fans at having failed to set a new record, their winning run has almost certainly propelled them into the Champions League next season and made them clear favourites for the title.

Sir Alex Ferguson would often refer to momentum as being instrumental to success. A winning streak can sweep teams to the title or snatch survival from the jaws of relegation. What constitutes a good streak is clearly dependent on the team, though.  Manchester United are currently on a five-match winning run: such form would certainly be outstanding for a relegation-threatened team, but is it common for a Champions League contender? This question is itself part of a broader one: what is form and how should we measure it?

In this blog I’m going to take a look at some of the statistics of winning streaks, investigating the characteristic length of winning runs in the EPL and how it varies for teams from the top to the bottom of the table.

How well do teams streak?


I started by taking every completed EPL season since 2000/01 and dividing the teams into bins based on their points total at the end of each season (0-40 points, 40-50, 50-60, and so on)[1]. For each bin, I measured the proportion of the teams in that bin that completed a winning streak, varying the length of the streaks from 2 to 10 matches.  For example, of the 54 sides that have finished on between 50 and 60 points since the 2000/01 season, 17 (31%) completed a winning run of at least 4 matches.  Runs were only measured within a single season – they do not bridge successive seasons[2]. The results are summarized in Table 1.


Table 1: The proportion of teams that complete winning runs of two games or longer in the EPL. Teams are divided into bins based on their final points total in a season, from 0-40 points (top row) to >80 points (bottom row).

The top row gives the results for teams that finished on less than 40 points. The columns show the percentage that managed a winning streak, with the length of the streaks increasing from 2 (left column) to >10 matches (right). Three quarters of the teams in this points bin put together a winning streak of at least two games. However, the proportion drops very rapidly for longer runs: only 14% completed a 3-match winning streak and only 7% a 4-match streak. The only team to complete a 5-match winning streak was Newcastle early in 2014/15 (and this was half of the total number of games they won that season).

As you'd expect, the percentage of teams that achieve a winning streak of a given length increases as you move to higher points bins. Every team that has finished with 60 points or more has completed a 3-match winning stream. However, fewer than a quarter of those that finished with less than 70 points completed a 5-match winning streak. In general, the proportion of teams that achieve a winning streak drops off very rapidly as the length of the streak is increased. 

The exception is the title-challenging teams (the bottom row in Table 1): the percentage in this bin falls away more slowly as the the length of the winning streak is increased. 27 of the 29 teams that finished with at least 80 points put together a 5-match winning streak, 13 completed an 8-match streak and 5 completed a 10-match winning streak. This is the success-generating momentum that Ferguson habitually referred to.

In his final 13 seasons (from 2000/01 to 2012/13), Man United put together 14 winning streaks lasting 6 matches or more; in the same period Arsenal managed only 5. United won 7 titles to Arsenal’s 2. For both teams, the majority of these streaks occurred in title-winning seasons. The same applies to Chelsea and, more recently, Man City. Only two title-winning teams have failed to complete a 5-match winning streak: Man United in 2010/11 and Chelsea in 2014/15. The median length of winning streak for the champions is between 7 and 8 games.

Leicester’s 4-match winning streak at the end of the 2013/14 season saved them from relegation. It was also an unusually long run for a team finishing on around 40 points - only four other teams have managed it. Was this a harbinger of things to come? A year later, during their title-winning season, their 5-match winning streak in March/April pushed them over the line.

The implications for form


Only the best teams put together extended winning runs: 40% of EPL teams fail to put together a three-game winning streak and 64% fail to win 4 consecutive games. Perhaps momentum - and the belief and confidence it affords - is only really relevant to the top teams? Does the fixture list throw too many obstacles in the path of the smaller teams? Every 3 or 4 games a smaller team will play one of the top-5 sides, a game that they are likely to lose. This may make it more difficult for them to build up a head of steam.

On the other hand, perhaps smaller teams are able to shrug-off their defeats away to Arsenal or Liverpool and continue as before. In that case, should we discard games against the ‘big teams’ when attempting to measure their form? And to what extent do draws interrupt, or in some cases boost, a team's momentum? These are all questions that I intend to return to in future blogs.

Unbeaten Runs


Finally, I’ll leave you with the equivalent table for unbeaten runs. While the typical length of unbeaten runs in each bins is about twice as long as winning runs, most of the conclusions above still apply.

Table 2: The proportion of teams that complete an unbeaten run of length 2 or longer in the EPL. Teams are divided into bins based on their final points total in a season, from less than 40 points (top row) to more than 80 (bottom).

---------------

Thanks to David Shaw for comments.

[1] The total number of teams across all bins was 320: 16 seasons with 20 teams per season.
[2] Note that the runs are inclusive - if a team achieves a 3-match streak it will also have achieved a 2-match streak.




Tuesday, 20 December 2016

Does January transfer spending improve results?

Last week the Sunderland chief executive, Martin Bain, warned that only "very limited" funds will be made available to David Moyes in the January transfer window (see here, here and here). Bain said that Sunderland are “not going to be able to spend to get out of trouble” and that "we have reached a point where there has to be a time where you don’t have that short-term hit to plug the holes in the dam".

The implication is that Sunderland have put their long-term financial health at risk in previous seasons by spending substantial sums in January in a last-ditch effort to retain their EPL status. While they have indeed survived their recent flirtations with relegation, is there any compelling evidence that winter spending actually improves results in the second half of the season? By out-spending their rivals, are troubled teams boosting their chances of staying up, or are they just using up previous financial resource that could be invested more carefully in their future? In this blog I’ll try to investigate these questions.

January spending and results improvement.


The goal is to establish whether there is any relationship between January transfer spending and an improvement in results in the latter half of the season. For each of the last six seasons, I calculated the gross January expenditure of every EPL team using data taken from transferleague.co.uk[1].  To measure the improvement in results for each team, I calculated the average number of points per game they collected in matches played either before or after January 1st in each season and took the difference (second half of the season minus the first).

Figure 1 below plots the change in points-per-game versus gross January expenditure for all EPL teams in each of the 2010/11 to the 2015/16 seasons (each point represents a team in one of those six seasons). On average, just under two thirds of EPL teams spent more than £1m in (disclosed) transfer fees in any given January window, with just over a third spending more than £5m and a fifth spending more than £10m. There are four clubs that spent more than £30m in January: Chelsea in 2010/11 and 2013/14, Liverpool in 2010/11 and Man United in 2013/14. The average change in points/game between the two halves of the season is close to zero[2] and there is no significant correlation with the level of spending.


Figure 1: Change in the average points-per-game measured before and after 1st January against total spending in the January transfer window for all EPL teams in each of the last six seasons. 

Not all teams will be looking for an immediate return on their investment in January. Some will be buying back-up to their first team or young players for the future. The teams that will certainly be looking for an immediate impact are those embroiled in the fight to remain in the EPL. In Figure 2 I’ve highlighted the relegation-threatened teams in each season. Specifically, this includes all teams that were in the bottom 6 positions in the table on January 1st, plus those that went on to be relegated at the end of the season (as you’d expect, most relegated teams were also in the bottom 6 in January)[3]. Teams that were relegated are coloured red; those that survived are blue. 

Figure 2: Change in the average points-per-game measured before and after 1st January against total spending in the January transfer window for all EPL teams (grey crosses) in each of the last six seasons. Teams marked by a square were in the bottom six of the table on 1st January; those in red were relegated, those in blue survived.
There are a couple of interesting things about this plot. First -- the majority of relegation-threatened teams see an improvement in their results in the second half of the season. I think this is just mean reversion: teams that underperform in the first half of the season are likely to do better in the second half. For example, over the last six seasons, teams in the bottom half of the table collected an average of 0.2 points/game more in the second half of the season than the first. The opposite is true of teams in the top half of the table: they tended to be an average of 0.2 points/game worse-off in the second half of the season. 

Second -- there is no significant correlation between spending and improvement in results for relegation-threatened teams. If we split them into two groups, those that spent greater than £5m in January and those that spent less, we find that 38% (6/16) of the high spenders and 55% (12/22) of the low spenders were relegated. This difference is probably not big enough to be significant. Raising the stakes higher – of the four relegation-threatened teams that spent more than £20m in January, three were relegated: Newcastle & Norwich last year, and QPR in 2012/13.

It seems reasonable to conclude that teams should resist the temptation to try to spend their way out of trouble: there is little evidence that it will pay off. It looks like Bain is being prudent in tightening the purse strings.

-----

[1] Note that for some teams it will be an underestimate as the transfer fee was never disclosed.
[2] This doesn’t have to be the case. For instance, there could be more draws in the first or second half of the season.
[3] The results don't change significantly if we selected relegation-threatened teams as being those within a fixed number of points from the relegation zone.

Friday, 2 December 2016

Playing in Europe does affect domestic results in the EPL

There’s recently been a bit of discussion in the media (e.g: Sky, Guardian) on whether participation in European competitions has a negative impact on an EPL club’s domestic performance. This is partly motivated by the significant improvements shown by Liverpool and Chelsea this season: after 13 games they are 10 and 17 points better off than at the same stage last season, respectively. Neither are playing in Europe this year. Leicester are demonstrating a similar trait, albeit in the opposite direction: they are now 15 points worse off than last season. For them, the Champions League seems to have been a significant distraction.

Numerous studies have demonstrated that there is no ‘hangover’ effect (see here and here) from playing in Europe. There is no evidence that EPL teams consistently perform worse in league matches that immediately follow a midweek European fixture. But what about the longer-term impact? Perhaps the mental and physical exertion of playing against the best teams in Europe manifests itself gradually over a season, rather than in the immediate aftermath of European games. If this is the case, we should be able to relate variations in an EPL team’s points haul from season-to-season to the difference in the number of European fixtures it played.

It turns out that there is indeed evidence for a longer-term impact. The scatter plot below shows the difference in the number of European games played by EPL teams in successive seasons against the change in their final points total, over the last 10 seasons. Each point represents a single club over successive seasons. For instance, the right-most point shows Fulham FC from the 08/09 to 09/10 season: in 09/10 they played 15 games in the Europa cup (having not played in Europe in 08/09) and collected 7 fewer points in the EPL. Teams are only included in the plot if they played in European competitions in one or both of two successive seasons[1]. The green points indicate the results for this season relative to last (up to game week 13); the potential impact of European football (or lack of) on Chelsea, Liverpool, Southampton and Leicester is evident. Chelsea's league performance from 2014/15 to 2015/16 is a clear outlier: they played the same number of Champions League games but ended last season 37 points worse off.
Effect of participation in European competitions on a team's points total in the EPL over successive seasons. Green diamonds show the latest results for this season compared to the same stage last season. Blue dashed line shows results of a linear regression. 

The blue dashed line shows the results of a simple linear regression. Although the relationship is not particularly strong – the r-square statistic is 0.2 – it’s certainly statistically significant[2]. The slope coefficient of the regression implies that, for each extra game a team plays in the Europe, they can expect to lose half a point relative to the previous season. So, if a team plays 12 more games, it will be 6 points worse off (on average) than the previous season. 

It’s worth noting that the CIES Football Observatory performed a similar analysis in a comprehensive report on this topic published earlier this year.  They found there to be no relationship between domestic form and European participation over successive seasons. However, in their analysis they combined results from 15 different leagues across Europe. So perhaps the effect is more pronounced in the EPL than other leagues? This recent article in the Guardian, citing work by Omar Chaudhuri, suggests that the effects of playing in Europe may be more pronounced in highly competitive divisions. The lack of a winter break may also be a factor: while teams in Italy, Spain and Germany enjoy several weeks rest, EPL teams will play four league matches over the Christmas period. 

Finally, an obvious question is whether we are simply measuring the effects of playing more games across a season. To test this, we should apply the same analysis to progress in domestic cup competitions. However, I’ll leave that to the next blog.


----------------------

[1]. The points along x=0 are teams that played the same number of European games in successive seasons (and did play in Europe both seasons). The only two teams that are omitted are Wigan and Birmingham City, both of whom played in the Europa League while in the Championship. Matches played in preliminary rounds are not counted.
[2] The null hypothesis of no correlation is resoundingly rejected.

Friday, 25 November 2016

Final Table Predictions for the EPL

In a previous post I looked at how the EPL league table evolves over a season, showing that we already have a decent idea of how the final league table will look after just a third of the season.

I’ve now taken that analysis a step further and built a simple model for predicting the total number of points each team will accumulate over the season (and therefore their final rankings). What follows is a short summary of how the model works; I've provided more technical detail at the end.

Season simulations


Each team starts with their current points total. I then work my way through the fixture schedule (currently 260 matches), simulating the outcome of each game. Results are generated based on the Elo rankings of each team – which I update after each simulated match – and the benefits of home advantage (scroll down to the last section for more details). At the end of the ‘season’, I tally up the final points totals for each team.

This process is repeated 10,000 times to evaluate the range of points that each team ends up on; I then make a final league table based on their averages. The probability of each team finishing the season as champions, in the top four or bottom three is calculated based on the frequency at which it occurs within the 10,000 runs.

Final table predictions 


Using all the results to date, the projected EPL table looks like this.

The box plots indicate the distribution of each team's points totals over the 10,000 simulated seasons. The green bars indicate the 25th to 75th percentiles and the dashed lines (‘whiskers’) the 5th to 95th percentiles. For example, in 50% of the simulations Man City finish on between 71 and 81 points and in 90% of the simulations they accumulate between 63 and 89 points. The vertical line in the middle of the green bars shows the median[1]. The numbers to the right of the plot show the probability of each team: 
a) winning the title (Ti);
b) finishing in the champions league spots (CL);
c) being relegated (rel).

You can see that the table is bunched into three groups: those with a decent chance of making it into the champions league, the solidly mid-table teams and the remainder at the bottom. Let’s look at each group in turn.

Top Group: This group contains Man City, Chelsea, Liverpool, Arsenal, Spurs and, if we’re being generous, Man United. These are the teams with a fighting chance of finishing in the top four. City, Chelsea, Liverpool and Arsenal are so tightly bunched they are basically indistinguishable: you can’t really predict which of them will win the league. However, there is a 93% probability that it’ll be one of those four. Spurs go on to be champions on only 6% of the simulations and United in less than 1%. Indeed, United finish in the top four only 17% of the time – roughly a 1 in 6 chance.

Middle Group: This group includes Southampton, Leicester, Everton, Watford and West Brom. The distribution of their points totals indicate that they are likely to collect more than 40 points, but less than 60. That makes them reasonably safe from relegation but unlikely to finish in the top four (last season, the 4th placed team – Man City – finished with 66 points). They can afford to really focus on the cup competitions (and for Leicester, the champions league).

Bottom Group: Finally, we have the remaining nine teams, from Stoke down to Hull. According to my simulations, these teams have at least a 10% chance of being relegated. The bottom 5 in particular collect less than 40 points on average and are relegated in at least a third of the simulations, with Sunderland and Hull going down more often than not. 

Next Steps


My plan is to update this table after each round of EPL games (which you can find here). Hopefully, we should see the table beginning to crystallize as the season progresses, with the range of points totals narrowing and thus the final league positions becoming easier to predict.

There is also plenty of information that could be added. The simulations know nothing about injuries and suspensions, future transfers, managerial changes and grudge matches. They also do not take into account fixture congestion and cup participation. I’m going to investigate some of these issues and incorporate anything that reliably adds new predictive information.


-------------------------

Specific Model Details


This section takes a look at what is going on under the hood in a bit more detail.

The core of the calculation is the method for simulating match outcomes. For each match, the number of goals scored by a team is drawn from a Poisson distribution with the mean, μ, given by a simple linear model:

There are two predictors in the model: X1 = ΔElo/400, the difference between the team's Elo score and their opponents', and X2 is a binary home/away indictor equal to 1 for the home team and -1 for the away team. Note that Elo scores are explicitly designed to be predictive of match outcomes. The initial Elo score for each team is taken from clubelo.com; after each simulated fixture the Elo scores are updated using the procedure described here.

The beta coefficients are determined via linear regression using all matches for the seasons 2011/12 to 2015/16, obtaining values β1 = 0.26, β2 = 0.71, β3 = 0.13. All are highly significant, as is the change in deviance relative to an intercept-only model. Running the regression on earlier seasons obtains similar results. 

How good are the match predictions?


A good way of answering this question is to compare the match outcome forecasts generated by this model with the probabilities implied by bookmaker's betting odds. There are a number of different metrics you can use to compare forecast accuracy, I’ve chosen two: the Brier score and the geometric mean of the probabilities of the actual match outcomes. It turns out the Poisson model and the bookies do equally well: they have identical scores for both metrics (0.61 for the Brier score and 0.36 for the average probability - consistent with what this analysis found).

The plot below shows that there is a strong relationship between the predicted probability of home wins, away wins and draws for the EightyFivePoints model and the bookmaker’s forecasts (note that I've 'renormalised' the bookmaker's odds such that the outcome probabilities sum to 1 for any given match). This makes me think that they’re doing something quite similar, with a few extra bells and whistles.

Comparison of probabilities assigned to ‘home win’, ‘away win’ and ‘draw’ by the Poisson model and those implied by bookmakers odds. All EPL matches from the 2011/12 to 2015/16 seasons are plotted.

One stand out feature is that draws are never the favoured outcome. This suggests that one of the keys to improving the accuracy of match outcome predictions is to better identify when draws are the most likely outcome. After all, more than a quarter of games end in draws.


------
[1] Which happens to be close to the mean, so there isn’t much skew.

Saturday, 12 November 2016

Elo Impact: Who are the EPL’s most effective managers?

Manager rivalry is one of the big themes of the season. Many of Europe’s most successful managers have converged on the EPL, sparking renewed and fierce competition between England’s biggest clubs as they battle on the pitch to achieve domestic superiority.  In the background there is another competition, one of a more individual nature. Guardiola, Mourinho, Conte and Klopp are seeking to establish themselves as the pre-eminent manager of their generation. As touchline galacticos, their rivalry mirrors that of Europe’s top players.

Success is often measured relative to expectation. Second place this season would probably be seen as a good finish for Liverpool, but not Man City. So Klopp and Guardiola will be judged against different standards. If Moyes guides Sunderland to a top ten finish he’ll win manager of the season.

For the same reason, it’s difficult to compare their track records. A manager may have won an armful of medals, but was it the result of years of sustained improvement or a few tweaks to an already excellent team? Can we compare the achievements of Wenger and Pulis, or Ferguson at Aberdeen and Ferguson at Man United?

To answer these questions we need an objective method for comparing the track records of managers over their careers. Not a count of the big cups in their cabinets, but a consistent and transferable measure of how much they actually improved their teams. In this post I’m going to lay out a simple method for measuring the impact managers have made at their clubs. I’ll then use it to compare the careers of some of the EPL’s current crop of talent.

Elo Scores


There is one measure of success that is applicable to all managers: to increase the number of games the team wins. The problem is that it is not easily comparable over time: a manager can move from a small club to a big club, or one league to another, and his win percentage will vary irrespective of the impact he had on each team.  However, there is a neat way of circumventing these issues, and that is to use the Elo score system.

Created by physicist Arpad Elo for ranking chess players, the Elo system has now been applied to a number of different sports, including the NFL and international football teams. The excellent site clubelo.com has adapted it for European club football. You can find all the details there, but here are the essentials: each team has an Elo score which varies over time as they win, draw or lose matches. The difference in scores between two teams is directly related to the probability of each team winning in a direct confrontation.

For example, Man United currently have an Elo score of 1778 and Barcelona 2013; the difference is 235 and under the Elo system this implies that Barcelona would have an 80% chance of winning the game (if played at a neutral venue). The full details of this calculation can be found here.

After two teams have played they will exchange points, with the exact amount being dependent on two things: the difference in their Elo scores before the game, and the outcome. For example, last weekend Man City drew 1-1 with Middlesbrough. As City were expected to win the game Middlesbrough gained 7.5 points and City lost the same number.

So how do we apply the Elo system to measure manager impact?

Manager Impact


We can assess the impact a manger has made by simply tracking the changes to the club’s Elo score since he took charge. I’ll refer to this as the manager’s Elo Impact. The neat part is that we can consistently monitor a manager’s record across multiple clubs by simply summing up all the changes to Elo scores over his career. Unlike win percentage, this works because the numbers of Elo points a team gains for a win is dependent on how superior they are relative to their opponent: in the Bundesliga, Bayern Munich receive far fewer points per win than Darmstadt 98.

Let’s look at a couple of examples. The two figures below show the Elo Impact of two managers across their careers: Alex Ferguson and Jose Mourinho (similar plots for Wenger, Guardiola, Klopp and Conte can be found here). For each manager, I’ve only included periods spent at UEFA clubs (omitting Wenger’s time in Japan, for example) and at clubs in the top two divisions of each country.

Figure 1 starts in 1978, when Alex Ferguson took over at Aberdeen, and ends with his retirement in 2013. The red line tracks the cumulative sum of the changes to his Elo score, bridging his move from Aberdeen to Manchester United in 1986.

Figure 1: the Elo Impact of Sir Alex Ferguson from 1978.

The first thing that strikes me is that his peak at Aberdeen – the 1983-84 season, when he won the Scottish league and European cup-winners cup – is almost level with his peak at Man United manager (his second Champions League and 10th EPL title in 2008). This implies that Ferguson’s impact at Aberdeen and United are comparable achievements. That’s not an unreasonable statement: Ferguson won 3 of Aberdeen’s total of four Scottish titles and is still the last manager to break the Old Firm hegemony. 
  
The striking thing about Mourinho’s Elo Impact (Figure 2) is that it is so much less volatile that Ferguson’s. Yes, the axis range is broader – Mourinho has had a lot of success in his career and his peak impact (at around 500) is substantially higher than Ferguson’s – but a quick estimate shows that Ferguson’s score fluctuates about 30% more. On closer inspection, this might be because Ferguson’s teams tended to win more of the big games but lose more frequently to weak teams than Mourinho’s (at least, until recently). However, this needs further investigation.

Figure 2: the Elo Impact of Jose Mourinho from 2004.

It’s worth emphasizing that the Elo score does not go up simply because trophies have been won, it does so if the team improves relatives to its peers. Jose Mourinho’s time at Inter is a good example of this. Despite winning the treble in his final season in 2010, Mourinho departed Inter having made little improved to their Elo score. This is because Inter were already the dominant force in Italy when he arrived, having won Serie A in each of the preceding three seasons. Put simply, it’s difficult to significantly improve the Elo score of a team that is already at the top. Guardiola’s time at Bayern Munich is another example.[2]

Who are the most effective managers in the EPL?


We can also use Elo Impact to rank managers. There is a question of how best to do this: by total impact (latest score), average impact over the career (score divided by total number of years in management), or by score this season. I’ve decided to provide all three, but have ranked managers by their total impact. The results are shown in the table below.

Total, average (per year) and 16/17 season Elo Impact scores for current EPL managers.

The top 6 are pretty much what you’d expect, with one very notable exception. Tony Pulis, who has never actually won a major trophy as a manager, leads the table. This is not crazy: Pulis has improved the standing of every major club that he managed (a plot of his career Elo Impact can be found here). In particular, over his two stints as Stoke City manager, he took them from a relegation threatened Championship team to an establish mid-table EPL team. 

I think that the example of Tony Pulis demonstrates one of the strengths of the Elo Impact metric – it is fairly agnostic as to where a team finishes in the league, so long as the team has improved. While we are naturally attracted to big shiny silver cups, some of the best work is being done at the smaller clubs. I fully acknowledge that repeatedly saving teams from relegation requires a very different managerial skillset to developing a new philosophy of football at one world’s most famous clubs; the point is that Elo Impact at least allows you to put two very different achievements on a similar footing. It’s a results-based metric and cares little for style.[1]

Guardiola is perhaps lower than some might expect, but then he only had a small impact on Bayern Munich’s Elo score during his tenure. A few successful seasons at City and he’ll probably be near the top of this table. Why is Wenger’s average impact so low? As this plot shows, he substantially improved Arsenal during the first half of his tenure, but has essentially flat-lined since the ‘invincibles’ season. Further down the table, Bilic's score has fallen substantially this season as West Ham have had a disappointing campaign so far. 

So what now?


I intend to develop Elo Impact scores for two purposes. First, I’ll track each manager’s scores over the EPL season to track who has had overseen the greatest improvement in their side. I’m happy to provide manager rankings for other leagues or individual clubs on request.  Second, as new managers arrive, I’ll look at their Elo track record to gain an insight on whether they’re likely to be a be success or not. 

It's going to be fascinating to see which manager comes out on top this season.

-----------------

Thanks to David Shaw for comments.


[1] Although you do gain/lose more points for big victories/losses.
[2] It is difficult to improve, or even just maintain, a team's Elo score once it rises above 2000. Few points are gained for winnings games and many are lost for losing them. Basically, the team is already at (or near) the pinacle of European football. For this reason I've made a slight correction to the Elo Impact measure: when a club's Elo score is greater than 2000 points, I've set the maximum decrease in a manager's Elo Impact to 10 points per game. Once the club's score drops below 2000, the normal rules apply.