2017 Draft: How do OHL Goalies Stack up Against NHL-Caliber Competition?

By: Mike Z

This article is being co-posted on NHLNumbers as well as on my own site, OriginalSixAnalytics.com. Find me @OrgSixAnalytics on twitter. All Junior stats used here were provided by @3Hayden2, the creator of www.prospect-stats.com. Please give him a follow and check out his great work. (Last, for those who do not know, I formerly wrote under the pen name of “Andrew Kerison”)

While both the NHL and Major Junior (WHL, OHL, QMJHL) can benefit from hockey analytics, the findings in each league are often quite different. The NHL is a league of parity, where the gap between a top player and an average one can be relatively thin. Junior, on the other hand, is the opposite. Top players like Mitch Marner, Matthew Tkachuk, and Brayden Point can often make an immediate impact at the NHL level, while many of their teenage competitors will never play an NHL game at all. The massive range of Junior talent is probably best illustrated by the fact that Connor McDavid was able to win the 2016-2017 NHL scoring race while still being young enough to play for his former Erie Otters team in last week’s Mastercard Memorial Cup!

This wide range of CHL talent has an outsized impact on player evaluation in one area in particular: goaltenders. The NHL draft already has a significant amount of randomness, and goalies are already likely the single hardest position to project NHL performance. Given the wide range of Junior shooting talent, quality of competition can have a disproportionate impact on certain goalies depending on their team’s particular schedule. If one goalie plays a substantial number of games against teams like the Erie Otters and London Knights, we shouldn’t expect him to have the same result as a peer with an easier schedule.

As such, in this article I will try to analyze these factors in an attempt to better evaluate this year’s goaltending prospects. Specifically, I will try to isolate how OHL goalies perform against the higher end of shooters at the Junior level.

NHL vs OHL: Shooting Talent Distribution

In the two tables below I have looked at the top 15 NHL and OHL shooters when ranked by their shooting %. All data is from 2016-17, forwards only, and I have restricted the samples to 5v5 to adjust for the impact of power play time. Both samples were drawn from players with >100 shots (NHL) and >70 shots (OHL) in order to remove outliers.

1. Top NHL

2. Top OHL

NHL data courtesy of puckalytics.com; OHL data from prospect-stats.com

As you can see, there is a substantial gap between the top 15 NHLers and OHLers. Rickard Rakell shot the lights out this year with a 19% NHL Sh%, and a full seven OHL shooters were at that level or higher. This disparity is highlighted even more from some general distribution statistics:

3. OHL v NHL Distrib Stats

NHL data courtesy of puckalytics.com; OHL data from prospect-stats.com

As you can see, this gap isn’t limited to just the top end of the spectrum. While the bottom quarter of shooters (25th percentile) aren’t too far apart between leagues, the gap grows at the median and then diverges significantly at the 75th percentile, where respective OHL and NHL shooters are almost 4.0% apart.

 OHL Shooting Talent – Team Level Gaps

The OHL is a league that has ‘juggernaut’ type teams, who will often dominate games except for when they are playing other top-ranked competition. Just look at the results from last week’s Memorial Cup, with two (Western Conference) OHL teams winning all of their games against the other league champions, sometimes by a wide margin, and eventually facing off in the final.

In order to determine performance against NHL-like shooters, in a perfect world we could isolate the shots coming from each individual skater against each goalie. Unfortunately, for this article (and based on what was most readily available), I have instead tried to isolate these dominant OHL teams, and considered that as a proxy for which players represent future “NHL-Caliber” shooting talent. Although this is a bit of an oversimplification, I think it will be an interesting analysis nonetheless. Even just looking at the NHL prospects on the Memorial Cup Finalist Erie Otters (Strome, Raddysh, DeBrincat, Cirelli, Foegle, Lodnia, etc.) – many of these ‘juggernaut’ teams are simply stacked.

The table below shows all OHL teams from the 2016-2017 season alongside their All Situation shooting % (all metrics in this article exclude empty net goals):

4. NHL Caliber Teams

OHL data from prospect-stats.com

As you can see above – I drew a (somewhat arbitrary) line at the 65th percentile to denote the ‘NHL Caliber’ shooting teams. An argument can be made to increase this cut off to maybe 11.0%, or 11.5%, but the tradeoff later on will be having enough teams included so there is still a useful sample for each goalie. Personally, I was surprised that this omitted some high-end, skilled teams like the Mississauga Steelheads, but they may have been held back due to a slow start to the 2016-2017 season.

As a whole, these teams shot at an average all situation Sh% of 11.7%, versus a league-wide average of 10.2%. For reference, the teams that were below the ~65th percentile shot on average at 9.3%, so a gap of ~2.4% between the included and excluded teams. From here on I will refer to these top teams as having ‘NHL-Caliber’ shooting talent.

Now – how do OHL goalies stack up when looking at shots faced from just these 8, very high-end shooting teams?

Unadjusted OHL Goaltender Performance (All Opponents)

The table below shows the top 20 OHL goalies when simply ranked by ‘raw’ All Situation Save% (list shortened for space). The table includes both all situation Sv% as well as 5v5, and it only includes goalies with over 500 total shots against. This chart also includes each goalies age, though I will caveat that the additional decimal place in some ages seem slightly off.

5. Top 20

OHL data from prospect-stats.com, goalies who are 18 years old and younger (e.g. never before eligible for the NHL draft) are highlighted in green

Looking at this table, you can see a wide range of OHL-level goaltending stats and ages. Highly touted prospects who are already signed to NHL contracts are older, coming out near the top – guys like Tyler Parsons (LDN – CGY) of USA 2017 World Junior fame, Dylan Wells (PBO – EDM), and OHL/CHL Goaltender of the Year Michael McNiven (OS – MTL)

The goalies that are eligible for the 2017 NHL draft are those born in 1999 and those after mid-September in 1998 (mostly 17 year olds). On this list that is only four individuals: Michael DiPietro (WSR), Matt Villalta (SSM), Stephen Dhillon (NIAG) and Luke Richardson (KIT). All four have reasonably strong results, especially at 5v5, where they have Sv% of 91.9%, 92.6%, 92.4% and 91.2%, respectively. It is worth noting that OHL rookie (16-year old) Jacob Ingham (MISS) has quite strong ‘raw’ results as well – but due to his age, he won’t be draft eligible until 2018.

Adjusted OHL Goaltender Performance (NHL Caliber Shooters)

As discussed earlier, not all OHL opponents are created equal – and depending on conference/division, goalies can face these opponents at quite different rates. In the table below, I have isolated the shots faced by these goalies strictly to those taken by the 8 teams identified earlier as shooting at an ‘NHL Caliber’ level. Further, due to sample size issues, goalies are included only if they have at least 200 shots against from these opponents in all situations. In general, some of these figures should be taken with a grain of salt, due to the smaller sample sizes resulting from this approach.

6.All Sit NHL v All

OHL data from prospect-stats.com. Note, if multiple goalies have the same rank, their Sv% is the same. ‘Appearances’ includes all games played (including going in a replacement), and is not equivalent to ‘starts’.

In the table above has two sections: how goalies’ All Situation Sv% ranks when isolating shots faced from NHL Caliber opponents, and also how they fared and ranked in Sv% against all OHL opponents. The table is sorted based on NHL Caliber opponents Sv%, and I have highlighted the 2017 draft-eligble goalies in green.

Starting at the top – Tyler Parsons (LDN) is clearly elite, and maintains his strong stats and #1 rank regardless of the opponent – I would expect him to be one of the most likely future NHL goaltenders in the OHL today. Michael DiPietro (WSR) is quite similar – but even more impressive is that he has performed at this level at the age of only 17 years old. Given his excellent results against opponents of all levels, and his starting role on the Memorial Cup Champion Windsor Spitfires – DiPietro definitely stands alone atop the ranks of this year’s draft eligible OHL goalies.

What is interesting in this analysis really comes down to ‘movers’ – Goalies who are ranked higher when looking at just NHL Caliber opponents than when using all opponents. Troy Timpano (ER) and Matthew Mancina (MISS) both show well in this data – both moving up 6 ranks when looking at just their strongest opponents. Luke Opilka (KIT) and Joseph Raaymakers (SSM) are two of the biggest movers, jumping 12 and 15 spots respectively in the NHL-Caliber ranking. In a perfect world, you’d like to see consistently strong results against both categories, like star goalies McNiven and Parsons – but strong results against top opponents are good to see nonetheless.

Due to the low samples we need to be cautious with reading too much into this in general, but especially for certain goalies – e.g. Scott Smith (PBO), Matt Villalta (SSM), Aidan Hughes (SAR), and Luke Richardson (KIT). If anything, the small sample suggests these goalies were in the back-up role, potentially being sheltered for part of the season, or facing weaker opponents in general. However, that doesn’t mean strong results should be discounted altogether. As an example, Villata still ranks #11 against NHL Caliber shooters, and Hughes jumps up 13 spots on the NHL-Caliber ranking; so directionally, it is still a positive indicator to see in general.

7. 5v5 NHL v All

OHL data from prospect-stats.com

Last – the table above is the same approach but instead showing 5v5 Sv% rather than all situations. I wont go into too much detail on the findings, but generally you see similar names near the top or having made big moves. In general, many people in the hockey analytics community prefer 5v5 Sv% as it takes out discrepancies from playing down a player. However, with 5v5 we have to be even more mindful of the sample size. In this chart I have kept the same list of goalies from the All Situations chart (where the cut off was 200 Shots Against vs NHL Caliber shooters), but at 5v5 some drop below 200, ultimately representing only a handful of games.

 Conclusion

In the end, this analysis is really more of a starting point, rather than a comprehensive look at this year’s first-time draft eligible OHL goalies. In the future, better ways to look at this analysis would be to possibly create an OHL-level QoC-Adjusted Sv% metric, which incorporates both strong and weak shooting opponent data but adjusts the result depending on which team/player took each shot, or alternatively to create a xGSAA metric that factors in both opponent quality and shot location. Hayden’s website already features shot location and Sv% data that is definitely worth a read. It is also worth mentioning that this analysis was entirely focused on one league, the OHL, and it is not at all a comprehensive look at 2017 draft eligible goalies.

While it is rare that teams will use a 1st round pick on a goaltender due to the overall risks involved, I would argue that this analysis definitely helps to substantiate that Michael DiPietro is the real deal. While it may take years for him to develop and become NHL-ready, I expect that whoever ends up taking him will find themselves glad they did – with an extremely exciting goaltending prospect to monitor for the years to come.

 

Wading into a Minefield: Why On-ice Sv% Stats May Not Be Completely Meaningless

This article is being co-posted on the new Puckalytics Blog as well as on my own site, OriginalSixAnalytics.com. Find me @OrgSixAnalytics on twitter.

 For those who don’t know – there is a hotly contested debate in the hockey analytics community about whether players are able to influence their teams’ save percentages. Many hockey analysts have looked into the subject, and some of our best and brightest have come out on opposite sides of the debate (e.g. Eric Tulsky, Garret Hohl, David Johnson and Kyle Dubas). As you can imagine, it is not a straightforward concept to pin down.

The core of the debate centers on the idea that for a statistic to be meaningful, it has to be repeatable – there needs to be evidence that players can consistently outperform/underperform on the metric over time. The current consensus is that there has been no evidence found to date to show that impacting on-ice Sv% is a repeatable skill. Instead, when you look across the league as a whole, what you find are outcomes that essentially resemble randomness.

Having been a goaltender my whole life, I never felt right about this conclusion. All of the analysis I have seen was sound, but my ‘gut’ (twitter explosion alert) told me that there must be some variance in the type of chances a goalie faces with different skaters in front of him. As a result, I have taken an attempt at reviewing past work on the subject, and trying to dig a little deeper – in particular, by trying to control for some additional variables that could be creating ‘noise’ in our findings.

Although I think I have reached some interesting conclusions, before getting into those results I’d like to openly invite the online ‘peer-review’ process. Only once others have pressure-tested these findings and concluded the work is (or is not) statistically meaningful will I be fully confident in the legitimacy of this outcome. However, hopefully this work is able to begin to move the conversation forward on the topic of whether or not relSv% and Sv%RelTM are relevant metrics in player analysis.

(For those who don’t know – Sv%RelTM represents a team’s save percentage when a particular player is on the ice, compared to how his teammates performed without him. It is expressed as a percentage above or below zero (much like CF%RelTM) – representing the amount he is above or below his teammates’ time-weighted average. Due to data availability, I have focused much of my analysis on Sv%RelTM, and to be clear, relSv% and Sv%RelTM are slightly different metrics. However, the findings shown here are almost exactly the same when using relSv%).

Prior Analysis of On-ice Sv% / relSv%

To start, I’d like to re-visit some of the historical work done on the subject. Eric Tulsky, a very well-regarded member of the hockey analytics community, wrote the initial ‘book’ on the subject. Tulsky demonstrated that on a league-wide basis, neither forwards nor defensemen are able to consistently display above/below average performance on the metric. He showed this in his 2013 article by using on-ice Sv% itself (rather than a relative stat), and by comparing a player’s ‘first three seasons’ of on ice Sv% to that player’s next three seasons. If this type of approach revealed a strong correlation, it would suggest players can consistently impact their team’s Sv% – for better or worse.

Tulsky’s analysis of forwards from 2007-2010 and 2010-2013 is shown below (his analysis of defensemen reaches largely the same conclusion):

Tulsky

(Source: Eric Tulsky)

As you can see, the plot appears to be scattered almost evenly and has an R2 value of less than 1%. This result suggests there is almost no correlation between past and future impact on on-ice Sv%, and that it is not a ‘repeatable’ skill.

However, following Tulsky’s work, there continued to be articles suggesting that certain NHL players do in fact have consistent impacts to on-ice Sv% relative to their teammates over time – arguably based on ‘anecdotal’ evidence. As a result, another well-known member of the community, Hockey Graph’s Garret Hohl, conducted an updated piece of analysis to show that – even when using a relative metric (here he used relSv%) – there remains limited evidence of it being a metric that is fundamentally able to predict future performance. I have copied his output below:

Holh

(Source: Garret Hohl)

As you can see, Garret used a similar methodology and demonstrated that the historical relSv% of a defenseman is not a reliable predictor of how that defender will do in the future – coming out at an R2 of only 2.6%.

Areas to Dig Further

However – as I suggested at the beginning – while these are absolutely valid findings based on the samples used, personal on-ice experience has kept this topic on my mind. Many writers often debate the wide range of factors that could be causing ‘noise’ in these results, such as:

  • Player usage
  • Quality of teammate
  • Quality of competition
  • Team-level systems
  • Quality of goaltender
  • Etc.

However, just because we have factors that could be causing noise in the findings does not mean we have a statistically valid statistic. Thinking through the lens of a goaltender, I hypothesized two specific factors I saw as major potential drivers of the noise in the relationship. They were:

  1. Changes to the starting goaltender a skater typically plays with
  2. Changes to the team a skater plays on

Intuitively, I think these factors make sense as things that could possibly impact on ice / relative Sv% numbers over time. For the first point, if a skater is playing in front of an elite goalie like Braden Holtby or Henrik Lundqvist – only specific, highly dangerous defensive lapses will actually turn into goals. However, if a skater is playing in front of Ray Emery – those same types of lapses may turn into incremental goals against. As a result, I wanted to investigate how Sv%RelTM’s predictive capabilities changed when a skater is consistently playing with the same starting goalie.

Secondly, when a player changes teams one can safely assume it would create a mess for this metric. For example, when Phil Kessel went from Toronto to Pittsburgh he changed not only the systems he played within, but also the individual teammates he is being compared against (on an innately relative metric). He also implicitly changed #1 – the starting goalie that he is most frequently playing with.

Controlling for Goaltender and Team Changes

As such, I set out to conduct a very similar analysis to what Garret did, while also attempting to control for the two factors above. I did so by limiting my sample to the following criteria:

  • Players included must have played for a single team for all four seasons studied (2010-2014)
  • Teams included must have had the same starting goalie for all four seasons
    • Starting goalie was defined as 50+ starts in full seasons and 20+ starts in the 2012-2013 lockout season
  • Players included must have played at least 500 minutes per season, and have been on the ice for at least 800 shots against

As you would expect, using all of the filters listed significantly reduced the size of my relevant sample. For example, only nine teams met the starting goalie criterion in this time period: SJS, MTL, CHI, NYR, DET, ANA, LAK, DAL and PIT. Further, across these nine teams, only 21 defensemen and 25 forwards (total n=46) played four consecutive seasons for one of those teams over this period. Of course, I need to point out that these sample sizes are definitely on the lower boundary of statistical significance. However, given the need to control for various factors, I think this was a necessary trade-off, and trying to increase sample size will be an important area to further investigate in the future.

As I move into the results in the next section, I have to say: they surprised me.

Save % Relative to Team – Defensemen only

Naturally, I think everyone considers supporting the goaltender and/or playing a major defensive role to be the primary job of just that – defensemen. This was my hypothesis going into the analysis, so, like Garret, I initially focused on just those players. However, as you can see from my chart below – the results were far from groundbreaking:

1 - Sv%RelTM Defense

 (Note – all data shown is even strength in order to remove the impact of PP and PK Sv%).

As you can see – there appears to be almost no relationship (R2 < 1%) between time periods in terms of a defenseman’s ability to impact his team’s on-ice Sv%. This finding is consistent when using relSv%.

One thing worth pointing out: this data has two relatively unique results in the top-left quadrant – Nick Leddy (CHI) and Cam Fowler (ANA). Both of these players had very poor Sv%RelTM results in 2010-2012 (roughly -2%), but improved significantly in 2012-2014 (1%). Given they both had their rookie seasons in 2010, one could argue their own development could have impacted their differences between periods. For interest’s sake, removing the two of them increases the R2 to approximately ~15%. Although this is interesting to note, this adjustment would need to be applied more broadly before we could draw conclusions from it.

Although this was ultimately less meaningful than I had hoped, my next thought was to add forwards to the sample – if nothing else, this would increase the sample size. This brings me to the next category:

Save % Relative to Team – Defensemen & Forwards

2 - Sv%RelTM Forwards &amp; D

Now, as you can see in the chart above, what I found here was both exciting and surprising. The exciting aspect is that there actually was some relationship – something otherwise unheard of in this type of analysis.  The surprising aspect of this came up when I asked myself: why are forwards the category of skaters who are able to show a sustained skill in Sv%RelTM, and not defenders? Before I had a good answer, I proceeded to do one final chart to test this unexpected result on its own:

Save % Relative to Team – Forwards only

3 - Sv%RelTM Forwards only

Looking at this chart is where I was blown away. By removing defensemen altogether (getting to a total n=25), you can see that the predictive value of Sv%RelTM reaches the highest found in any analysis I have reviewed – an R2 of 41% (or 38% for the same players when using relSv%). Although this isn’t a massive correlation, it is within the same range that xG and CF% fall into in terms of predicting a team’s future GF% over a season – metrics that are both widely regarded as valid for forecasting future outcomes.

Explaining These Findings

Now – as mentioned – the most difficult question here is: why are we getting this result? Why would a defenseman who fits the same team/goaltender-controlled data as the forwards sampled not be able to repeatedly impact his team’s on-ice Sv%? I certainly can’t say I know the answer, but one hypothesis that came up when discussing this with David Johnson: defensemen often play generalist roles, while forwards more often play specialized ones. For example – a 1st forward line is typically a scoring line, and a 3rd is typically a shutdown line. On the other hand, defensemen often get paired with their ‘opposite’ – having an offensively-minded defensemen paired with a stay-at-home one. I think these hypotheses intuitively make sense, however, the major drivers of these results remain open for interpretation.

Cross-team Applicability

One last question that will likely come up: given players can only consistently demonstrate Sv%RelTM impact when on a single team with a single starting goalie, does that mean it is not a skill transferable to other teams? My own (short) answer to this would be ‘no’. The relationship shown here for forwards indicates to me that it is in fact a skill, whether or not it would be displayed to the same degree on a different team or with a different goalie.

Much like CF% has significant team-level and player-level components – and won’t necessarily be directly transferrable between teams and systems – we still think of players as being strong/weak at driving shot-attempt differentials. I think the same logic can be applied here. But again – I am very curious about others’ thoughts.

Conclusion

In the end – this has been a hotly debated topic for quite a long period of time, and I doubt my work today will stop people from having different views on it. Although the analysis I have shown reaches a particular conclusion, I personally will not rely too heavily on this finding until it is broadly tested by the analytics community. However, given the desperate need in hockey to find additional ways to evaluate a player’s defensive contribution, hopefully this analysis helps us all continue to learn more about the merits of relSv% / Sv%RelTM as metrics for player evaluation. I look forward to anyone’s feedback or thoughts, @OrgSixAnalytics on twitter or at OriginalSixAnalytics [at] gmail [dot] com.