2015 NWSL Season

The navy dot is the actual value, the gold line and key are the model's reference where one applies, and the slate ticks are the cutoffs between read labels listed under each metric.

Home Edge

Percentage of games won by the home team - percentage of games won by the away team

No Edge: under 2% · Slight Edge: 2% to 6% · Clear Edge: 6% to 10% · Fortress: 10% and up.

Elo Value

Number of Elo rating points that 1 goal is worth. A team this many Elo ratings points better than another would be expected to defeat their opponent by 1 goal at a neutral location.

Scoring Tilt

Average home goals - average away goals per game. The gold line represents the expected home goals edge according to the scoring value of a point of Elo and the number of Elo rating points home advantage is worth.

Road-Tilted: under -0.8 · Lean Road: -0.8 to 0 · Lean Home: 0 to 0.8 · Home-Tilted: 0.8 and up.

The navy dot is the actual value, the gold line and key are the model's reference where one applies, and the slate ticks are the cutoffs between read labels listed under each metric.

Luck Spread

Average standard deviation of the difference between each team's actual points total and their simulated average points total. The gold line is the standard deviation the model expected from random chance alone.

Subdued: under 4.41 · As Expected: 4.41 to 6.06 · Elevated: 6.06 to 8.26 · High Variance: 8.26 and up.

Average Finish Error

Average gap in the rank of the final standings between where each team finished in the standings and where the model projected them to finish.

Pinpoint: under 1.48 · Close: 1.48 to 2 · Off Target: 2 to 2.61 · Well Off Target: 2.61 and up.

Biggest Overachiever

The team whose record finished in the highest percentile within its own simulated range of outcomes. The gold line indicates where the top team in a league of 9 teams typically lands.

Biggest Underachiever

The team whose record finished in the lowest percentile within its own simulated range of outcomes. The gold line indicates where the bottom team in a league of 9 teams typically lands.

Season Outliers

Number of teams that finished either above the 95th percentile or below the 5th percentile. Even with a well-calibrated model, 10% of teams are expected to land in the extremes of their distributions.

Minimal Outliers: under 0.45 · As Expected: 0.45 to 1.08 · Several Outliers: 1.08 to 1.98 · Many Outliers: 1.98 and up.

Unexpected Playoff Teams

Number of teams that actually made the playoffs but were not in the model's projected field.The gold line represents how many playoff teams the simulation missed on average.

Fewer Than Expected: under 0.94 · As Expected: 0.94 to 1.57 · More Than Expected: 1.57 to 2.51 · Far More Than Expected: 2.51 and up.

The navy dot is the actual value, the gold line and key are the model's reference where one applies, and the slate ticks are the cutoffs between read labels listed under each metric.

Gini Index

Measures how evenly points were distributed across the league. Lower scores indicate there was less variability across the actual point totals .Higher scores are an indication of more imbalance in the league with a few unusually strong or unusually weak teams.

Highly Balanced: under 0.08 · Balanced: 0.08 to 0.15 · Top-Heavy: 0.15 to 0.24 · Concentrated: 0.24 and up.

Noll-Scully

Measures how more times spread out the final standings were in reality compared to standings that were generated by flipping a coin for each game. The gold line at 1 indicates as much variability in the standings as s coin flip. Above it, talent gaps in team composition stretched the standings more than the variability produced by luck alone. Below it, the league was more competitive than coin flips alone would yield.

Compressed: under 0.9 · Coin-Flip Spread: 0.9 to 1.15 · Moderate Separation: 1.15 to 1.45 · Stratified: 1.45 and up.

Interquartile Edge

The probability that the team in the 75th percentile of the league's Elo ratings would defeat the team in the 25th percentile at a neutral site. The higher the probability, the larger the gap between the top teams and bottom teams in the league.

Even: under 58% · Competitive: 58% to 65% · Tilted: 65% to 73% · Lopsided: 73% and up.

Best vs. Worst

The probability that the team with the highest Elo rating would defeat the team with the lowest Elo rating at a neutral site.The higher the probability, the larger the gap between the top teams and bottom teams in the league. The gold line represents how high this probability is expected to be in a league of this size. A blue dot to the right indicates an unusually dominant team or an unusually weak team.

Tight: under 65% · Clear Edge: 65% to 78% · Lopsided: 78% to 88% · No Contest: 88% and up.

Close Games

Actual percentage of games decided by 1 goal or fewer. The gold line indicates the proportion of close games you would expect after considering the Elo ratings of the involved teams and the scoring value of a single Elo point.

Mostly Decisive: under 25% · Some Drama: 25% to 38% · Often Close: 38% to 50% · Nail-Biters: 50% and up.

Blowouts

Actual percentage of games decided by 3 goals or more. The gold line indicates the proportion of blowouts you would expect after considering the Elo ratings of the involved teams and the scoring value of a single Elo point.

Rare: under 10% · Occasional: 10% to 20% · Frequent: 20% to 30% · Routine: 30% and up.

The navy dot is the actual value, the gold line and key are the model's reference where one applies, and the slate ticks are the cutoffs between read labels listed under each metric.

Brier Score

How close the pregame win probabilities aligned with the actual game result, averaged over the entire season. Picks that were both confident and correct are rewarded the most, while incorrect predictions and/or uncertain predictions are penalized more. Lower scores are better. 0 indicates every game was perfectly predicted with maximum confidence. For sports with where wins and losses are the only outcomes, Brier scores less than 0.25 indicate the model did better than random predictions. For sports where ties are possible, 0.67 is the cutoff for which the Brier score must be lower to beat random predictions. The lower the Brier score, the easier and more accurately the games were to predict.

Highly Predictable: under 0.56 · Predictable: 0.56 to 0.62 · Hard to Predict: 0.62 to 0.66 · Coin-Flip: 0.66 and up.

Matchup Imbalance

How lopsided the matchups were on paper, calculated by averaging the difference in the win probabilities divided by the sum of the win probabilities. Lower scores indicate more parity in the league. 0 indicates that every game was a 50-50 toss-up between evenly matched teams. 1 means every game pitted a heavy favorite against a huge underdog.

Evenly Matched: under 0.12 · Slight Separation: 0.12 to 0.22 · Wide Gaps: 0.22 to 0.34 · Lopsided: 0.34 and up.

Strangeness

How extreme the actual final standings were compared to what the model expected. A value of 0 would indicate that the simulated results perfectly matched the actual standings. A score of 1 indicates that the actual results were scattered around the projections by exactly the variance the model predicted. (i.e. The projections missed the actual standings by 1 standard deviation on average. Scores larger than 1 mean the season was wilder to forecast with more overperforming and underperforming teams than expected. Scores less than 1 indicate the season was more predictable.

Tighter Than Modeled: under 0.85 · As Expected: 0.85 to 1.15 · Volatile: 1.15 to 1.5 · Chaotic: 1.5 and up.

Repeatability

How closely the order of the final standings aligned with the ranked preseason Elo ratings as calculated by the Spearman rank correlation. Larger values indicate that the preseason Elo ratings were a strong predictor of the team's final rank in the standings while values near 0 mean.

Reshuffled: under 0.3 · Weak Alignment: 0.3 to 0.6 · Strong Alignment: 0.6 to 0.85 · Closely Aligned: 0.85 and up.

Upset Rate

The percentage of the time the underdog won the game in reality. The gold line measures how often the model expected the underdog to win when considering the outcome probabilities. A blue dot to the right of the gold line means upsets were more prevalent than left. A dot to the left indicates that favorites won more often than expected.

Chalk Held: under 30% · As Expected: 30% to 40% · Upset-Prone: 40% to 46% · Bedlam: 46% and up.

Clear Favorite Upset Rate

The percentage of the time the underdog won the game in reality, but only including games with a clear favorite, one at least 60% likely to win once a draw is set aside. This metrics ignores games that could be considered toss-ups. The gold line is how often the model expected these heavy favorites to lose. A dot to the right means the big favorites were upset more than expected; to the left means they took care of business.

Favorites Held: under 22% · As Expected: 22% to 32% · Shaky Favorites: 32% to 42% · Bedlam: 42% and up.

The navy dot is the actual value, the gold line and key are the model's reference where one applies, and the slate ticks are the cutoffs between read labels listed under each metric.

Probability calibration

An all-in-one chi-square test of the model's probabilities. Games are sorted into bins according to how confident the model was. Within each bin, the predicted number of wins, losses, and ties are compared against the actual counts. The p-value of this test is plotted below. Values above 0.05 indicate that the model's probabilities are well-calibrated for the season.

Clear Misfit: under 0.01 · Misfit: 0.01 to 0.05 · Borderline: 0.05 to 0.1 · Well Calibrated: 0.1 and up.

Calibration slope

Checks whether the spread of the model's probabilities is correct. Each probability is turned into their log-odds. A line is fit that predicts the actual results from the log-odds. The slope of this line is the calibration slope.A slope of 1.00 is perfect, indicating that the model's confidence matches the actual outcomes of the games. Slopes less than 1.00 indicate overconfidence. The model assigned higher probabilities to the favorites than was appropriate given the matchup, causing more losses by favored teams. Slopes greater than 1.00 indicate under-confidence. The model is more uncertain about too many results, assigning them lower probabilities when in reality the favorites should have had higher probabilities.

Overconfident: under 0.85 · Calibrated: 0.85 to 1.15 · Underconfident: 1.15 and up.

Calibration error (ECE)

Calibration error (ECE) checks whether the model's probabilities mean what they say. ECE is calculated as the average weighted gap between the model's stated chances and how often the predicted outcome actually occurred. The smaller the ECE, the more confident we are about the model's probabilities. The gold line represents the noise ceiling, which is the amount of error that luck can produce on its own even if the probabilities are perfectly calibrated. Values below the noise ceiling indicate that the model's error is no larger than chance alone would produce. Values above the noice ceiling point to a genuine calibration issue since the error is larger than luck could explain.

Well Within Noise: under 0.08 · Within Noise: 0.08 to 0.17 · Above Noise: 0.17 to 0.33 · Well Above Noise: 0.33 and up.