Kukla's Korner

Kukla's Korner Hockey

Sometimes The Numbers Don’t Add Up

from Steve Simmons of the Toronto Sun,

On the night I first began to question advanced statistics in hockey, the stats man who sits a few seats down from me in the press box began regurgitating the game in numbers.

Mikhail Grabovski, he said, was the best Leaf that night. According to the numbers, Jay McClement was the worst.

About an hour earlier, when a colleague asked for advice on who to pick as his three stars for the next day’s newspaper, we both bypassed Grabovski, neither of us liking his rather singular game that night, and talked about the value of McClement, who had been particularly strong both defensively and killing penalties.

When I asked the stats man about the discrepancy between what we’d seen and what the numbers showed, he answered: “Sample size.”

That always seems to be the answer when the numbers don’t match what a discerning eye can see....

The statistics indicate Crosby had a fine playoffs. Crosby, himself, would disagree with the numbers. The stats people will tell you the game must adjust to the statistics but, really, the stats need to adjust to the game.

The game hasn’t changed all that much, other than speed and length of shift. The voices of analytics haven’t invented a new game, only a new way to look at it.

There is a place for what they do — just not a defining one. The game, through these eyes, is too free-flow, too incidental and accidental, too promiscuous to be naturally or easily analyzed with math.

more

Filed in: NHL Teams, NHL Talk, | KK Hockey | Permalink
 

Comments

J.J. from Kansas's avatar

Tyler Dellow tackled this well already

http://www.mc79hockey.com/?p=7065

Steve’s opinions on advanced stats are what they are, but the fact that he simply invents facts out of mid-air (both the JVR thing and the part about the Kings’ goal streaks are flat-out wrong) is troubling.  Steve Simmons no longer writes anything worth linking.

Posted by J.J. from Kansas on 05/21/14 at 08:17 AM ET

Avatar

Steve Simmons either left something out in this bit (can’t bother to read the rest of it), or the guy he was talking to is a moron. McClement has horrible numbers because he gets the crap zone starts, plays on the PK (hard to have possession numbers when you are down a man and all you are trying to do is clear it away), and thus vs. top competition. That doesn’t make him a bad player who had a bad night. That makes him an invaluable role player like a Malhotra, Talbot, Gordon etc.

It’s all about context. Looking at numbers plainly is just stupid.

Posted by redwingshomersLOL on 05/21/14 at 08:48 AM ET

Nathan's avatar

The funny thing is, I would be surprised if any more than a few straggler teams (looking at you, Edmonton… you too, Islanders) aren’t already employing at least a few folks in their front office to do statistical research. There’s too much money at stake. Yes, hockey is different than baseball in some ways that make it tougher to quantify, but the money is still there, and there are still things to be measured. It’s worth a relatively small investment. I would guess most of these teams have some proprietary metrics that they use when assessing their own players, potential free agent targets, draft targets, trade targets, and even for advance scouting on upcoming opponents.

Posted by Nathan from the scoresheet! on 05/21/14 at 08:58 AM ET

Primis's avatar

Here’s where the whole problem is IMO:

There’s a subset of fan that wants to use Advanced Stats as the bulk of whatever argument they want to make.  Advanced stats are there to complement and supplement the rest of the data, not trump it.

And until some individuals get that through their thick skulls, Advanced Stats aren’t going to have an easy time.  You can’t get b y on just Corsi, or Off. Zone Starts, or whatever stat you pick.  Advanced Stats are meant to be use din conjunction with more traditional stats AND the “eye test”.  A combination of the 3 can get you a good idea of what’s going on in most cases.

Otherwise, if you just cherry-pick stats you can make arguments that Johan Franzen and Brendan Smith were two of the 3 most-valuable players in the entire NHL because their ratio of Def. Zone Starts vs Off. Zone Finishes happened to be in the Top 3.  Does that tell the whole story?  Of course not.  It’s simply a puzzle piece, and it may in fact be an unimportant one in the grand scheme of things,

But that’s the rub:  the only people that fanatical over Advanced Stats are either a) trying to cite them to back a claim that no other data supports, or b) have a vested interest in the continued dissemination of said stats.

Posted by Primis on 05/21/14 at 10:01 AM ET

Avatar

Posted by Primis on 05/21/14 at 10:01 AM ET

Bing, Bang…..BOOOOOOOOOM!!

Posted by redwingshomersLOL on 05/21/14 at 10:04 AM ET

J.J. from Kansas's avatar

I don’t think the small subset of fans who inappropriately use advanced stats is the problem.  I think the large set of people who dismiss advanced stats users as though they are constantly cherry-picking stats which support a narrative are the problem.

The people who say Jay McClement is a better center or more valuable than Mikhail Grabovski are the problem.

I have yet to find an advanced stats author who says Johan Franzen or Brendan Smith are among the most-valuable players in the entire NHL. I constantly see dumb stuff like the McClement/Grabovski discussion brought up as fact though.

Posted by J.J. from Kansas on 05/21/14 at 10:08 AM ET

Avatar

I think the large set of people who dismiss advanced stats users as though they are constantly cherry-picking stats which support a narrative are the problem.

But that’s what most of them do, cherry-pick stats to support their narrative.  That’s what Lambert does in practically every article.  They celebrate Toronto’s collapse as proving their point, when the eye test did the exact same thing all year long.  No one watching that team could believe they were winning so much.  They were getting great goaltending until the Olympic break, so they were winning.  When they stopped getting great goaltending, they fell in the standings.  Common sense.

Then the stats geeks dismiss the Avalanche’s similar numbers because they will “eventually return to the norm or mean. Just not this year.”  Well, duh, eventually the Avalanche will have trouble if they keep letting their goalie get bombarded, just like Toronto’s goalies did.  Maybe it will be next year, or the year after or the year after that.  It might be October. 

This is how stats geeks fail.  They can’t or won’t or refuse to come to grips with the goalie as the great equalizer, as everyone who has ever watched hockey should know.

Simmons is right:

Not all shots on goal matter. Not all possession is meaningful puck possession. Not all faceoffs won will result in possession. Not all faceoffs lost end up with bad results. ...

And, as others have said, Simmons is right again:

There is a place for what they do — just not a defining one.

Posted by jkm2011 on 05/21/14 at 10:43 AM ET

J.J. from Kansas's avatar

No one watching that team could believe they were winning so much.

“Good thing the Leafs don’t play in the CHL. The CORSI hockey league. They’re doing just fine in NHL, though.” - Steve Simmons, 29 October, 2014

https://twitter.com/simmonssteve/statuses/395375707821309952

Then the stats geeks dismiss the Avalanche’s similar numbers because they will “eventually return to the norm or mean. Just not this year.”  Well, duh, eventually the Avalanche will have trouble if they keep letting their goalie get bombarded, just like Toronto’s goalies did.  Maybe it will be next year, or the year after or the year after that.  It might be October.

So you’re telling me it’s solely the eye test that tells you the Avalanche let their goalie get bombarded?  That’s cool, but I don’t have that kind of time to dedicate to watching the Avs all the time.

This is how stats geeks fail.  They can’t or won’t or refuse to come to grips with the goalie as the great equalizer, as everyone who has ever watched hockey should know.

People who are seemingly against using the stats properly seem to consistently do two things well

1. Create strawman arguments about what “stats geeks” are and aren’t saying about any specific topic.
2. Somehow magically come to the same conclusion those same stats geeks came to as though it’s actually intuitive.

There are seriously people who think Semyon Varlamov is truly a .927 sv% goalie who will continue to do that forever.

I’ll go back to Dellow on what it is the stats geeks are actually saying about this:

The difference between Steve and the analytics guys is that we acknowledge that there will be luck. Skill will usually win out but every year, there will be a couple of teams with whom it doesn’t. Steve sees it and assumes it means that something caused it.

When we said the Leafs weren’t going to make the playoffs, what we were really saying is that they were depending on lightning to strike in the same place two years in a row and that that wasn’t much of a plan. If Colorado doesn’t make massive improvements next year, they’re going to be relying on the same thing. Someone’s going to be lucky next year and get a big year from an unexpected goalie or have pucks go in. I don’t know who it will be but being the team that got lucky this year doesn’t tell me anything.

The defining role for analytics in hockey should be pretty obvious. While Mr. Simmons was doing a victory lap in October about how well the Leafs were doing (a position he later claimed he never took when he started telling everybody in March that he knew all along the Leafs were going to miss the playoffs), actual analysis was showing that the Leafs have a bad system. Additional analysis shows that they make personnel decisions based on bad information.

Meanwhile, more and more information keeps coming out from the actual good teams in the league (like Chicago) that indicates they’re embracing the use of statistical analysis to aid them in their decision-making (though not entirely drive it, as I’m sure some simp is sure to believe I implied).

Simmons is a very good storyteller and his audience seems to have a very short memory. They go to him to be entertained with narratives. There’s nothing wrong with that, but the fact of the matter is that more and more people than ever are going to guys like Mirtle to get their information because his information is simply better. It’s more-accurate and stands up to scrutiny better.

There will always be a place for narrative-weavers like Simmons, but the more he tries to chisel out a place for himself by knocking holes in more-rigorous and useful analysis, the more he limits himself.

Posted by J.J. from Kansas on 05/21/14 at 11:04 AM ET

Avatar

maxthesilent ‏@maximustacitus Oct 29
@arcticicehockey @simmonssteve Not all goals are meaningful not all points are game changing - no problem with those though?

Posted by redwingshomersLOL on 05/21/14 at 11:28 AM ET

Avatar

So you’re telling me it’s solely the eye test that tells you the Avalanche let their goalie get bombarded?  That’s cool, but I don’t have that kind of time to dedicate to watching the Avs all the time.

And that, right there, is The Admission (tm).

The stats guys often use stats as a replacement for actually watching the games, and as their central point of rebuttal towards GMs who actually do watch the film and the games who make decisions contrary to what the numbers allegedly infer.

The thing that obliterates the reliability of present day amateur analysis isn’t bad stats or a lack of time though… it’s bias.  That was true before advanced stats existed, and it’ll be true when the fifteenth generation of them crawl out of the amorphous ooze.

Stats are helpful, but stats without the ability or willingness to apply them free of bias are useless… and it’s the bias that kills analysis, both at the professional and amateur level.

That said, of the four major sports hockey will continue to be the least able to be helped by stats due to the nature of the game compared to the others.  It is too fluid, too random, and too interdependent.  There’s some good to be had in exploring the statistical possibilities in the years to come, but I think a lot of people are pushing way too many chips in regarding how useful these stats will actually be, largely because they see the aid they’ve provided in the NFL, baseball and basketball.

And not for nothing, but advanced stats in football and basketball are new enough that they’re just starting to run into some of the repeatability and forecasting issues which dog most crafted stat formulas.

At the end of the day, the truly elite expert observers of any sport don’t necessarily need to have the individual activities graphed out for them because as they watch the game happen their eyes assemble all of that information and they process it on the fly.  They are chefs who don’t need to read a recipe to know how much Cumin to add to the chili.

For the rest of us, though, we need the cheat sheets.

Posted by HockeyinHD on 05/21/14 at 11:30 AM ET

J.J. from Kansas's avatar

At the end of the day, the truly elite expert observers of any sport don’t necessarily need to have the individual activities graphed out for them because as they watch the game happen their eyes assemble all of that information and they process it on the fly.

Yeah and the truly elite composers can put a symphony together by ear.

They are the exception.  The problem is that most people who poo-poo the advanced statistical analysis aren’t the chefs who know how much cumin to add to the chili, they’re the regular people who burn Ramen.

Posted by J.J. from Kansas on 05/21/14 at 12:12 PM ET

SnLO's avatar

Posted by HockeyinHD on 05/21/14 at 11:30 AM ET

It’s a very rare occasion where I agree with your perspective, analysis and conclusion. This is one of those times.
I must also admit, I find it amusing that you make such a good case for the insignificance of hockey stats while previously you so often refer to stats to support your own arguments. I can’t recall reading anyone ever accusing you of watching a hockey game; if anything, it is that you need to watch some games. Good post though.

Posted by SnLO from beyond the M-1 on 05/21/14 at 12:20 PM ET

SnLO's avatar

...they’re the regular people who burn Ramen.
Posted by J.J. from Kansas on 05/21/14 at 12:12 PM ET

If they’re burning ramen, I’d argue they are mediocre people. Regular people can at least follow directions and not burn noodles in boiling water in less than 3 minutes.

Posted by SnLO from beyond the M-1 on 05/21/14 at 12:27 PM ET

redxblack's avatar

Cumin in chili? Gross.

Posted by redxblack from Akron Ohio on 05/21/14 at 12:31 PM ET

awould's avatar

Cumin in chili? Gross.

Crazy talk. It’s necessary. I just made the greatest pot of chili ever made and I used cumin. Also used turmeric.  Expand your horizons.

On the topic, it’s laughable that people are actually accusing the stats side of applying their stats with bias to frame a narrative, but those “expert eyes” who watch games and take it all in have no bias. It’s absurd. Of course, both sides can be guilty of bias and often are, but I think the average and even above average hockey-knowledge fan is biased all the time. Even the high-level experts. You think Babcock plays Cleary because he saw something in his Corsi numbers? Or do you think he’s got some sort of positive bias about Cleary’s impact on the ice that is not borne out by his actual production and effectiveness? He is biased towards Cleary for some reason.

The word “intangibles” is often used to mask bias. Doesn’t mean it’s always used that way, but when a coach or journalist is trying to defend a player who isn’t cutting it, that’s usually a word that is trotted out to off-set the lackluster numbers.

“He’s got those intangibles, he goes to the hard spots, he’s a glue guy, heart of the locker room.”

These things are opinions of a player and opinions are couched in bias.

Stats are useful. So is watching the game. Using stats to check what your eyes see is good. So is using stats to find things your eyes might’ve missed.

Posted by awould on 05/21/14 at 01:29 PM ET

Avatar

SO MANY STRAWMEN.

No “stats guy” worth his or her salt would tell you that stats tell the whole story, even the advanced stats. They would agree that hockey is too free flowing, and too little of the game is tracked statistically, to ever get a complete picture. They would likely agree that advanced stats in hockey probably will not ever reach the same comprehensiveness or reliability of those in baseball, football or basketball.

What the stats guy WOULD say, though, is that just because they aren’t perfect doesn’t mean they can’t dramatically enhance our ability to understand the game and expand our toolkit to evalulate individual players.

Furthermore, by giving us actual hard data, we can test our theories over time. Does a player’s Corsi today tell us anything about a player’s ability to positively or negatively affect outcomes in the future? (It does.) Does a team’s Corsi help us predict a team’s future winning percentage better than their past record? (Yes.) Can a skater influence his goaltender’s save percentage or his team’s shooting percentage while he’s on the ice? (Maybe, but not very much.) We look at these numbers, year after year after year, to see what things turn out to be valuable and what things turn out to be noise.

As the information gets more sophisticated, we can use the data to ask more specific questions like “should Team X be more aggressive after a faceoff loss in the offensive zone?” or “is Defenseman Y better than Defenseman Z at forcing teams to dump the puck in rather than carry it in?”, stuff that simply watching the games probably would miss. Numbers and math will never tell the whole story, but every year they are helping us to analyze and understand more and more aspects of the game that we never would have thought possible even a decade ago.

And then we can track these things over time and see how they perform, see if they have any predictive power, and see how much of an impact they have. If they work, they get accepted into the body of knowledge. If they fail, they get revised or rejected.

Now, in the vast majority of cases, the “advanced stats” are simply going to confirm what is already obvious. Traditional stats say Crosby is elite. The eye test says he’s elite. Advanced stats say he’s elite. Most of the time, advanced stats are simply going to confirm what you already know.

The real value in them, though, is they help you pick up some things that you might not be able to detect just by watching, or help you answer questions that might not be clear just by looking at the traditional stats, or even help you find some truths that might otherwise be counterintuitive. And in a league where lucky (or bad) bounces and hot (or cold) streaks happen all the time, advanced stats can help you sort out which players and teams are most likely to be “for real” and which are most likely to see their fortunes reversed.

Advanced stats correctly pointed out that Toronto had a broken system and that, if the goaltending fell even to average, they would get slaughtered. Advanced stats correctly identified Los Angeles as a lurking giant the year they won the Cup as an 8 seed. Advanced stats correctly predicted Minnesota’s collapse a few years ago after they roared out of the gate in the first few months of the season. And if Colorado can’t figure out how to keep the puck out of its own end next year (they’re a young team, granted, so they could still improve a lot in this area), advanced stats suggests there’s a good chacne they’ll fall like a rock down the standings. We’ll see, but so far the track record on this is pretty good.

Advanced stats will also help you pick out guys that the eyeball test might miss. Clarke MacArthur, for example, is a possession whiz year after year after year. Advanced stats love the guy. Does that mean he’s an elite hockey player? No, it does not, and you’d still rather have a guy who bangs in 80 points per year with middling possession numbers than MacArthur’s 50ish points per year with great possession numbers. But advanced stats can tell you that MacArthur brings a lot more to the table than his “traditional” statistics suggest, and teams that can identify these guys using advanced stats and choose them over guys with similar traditional numbers but less impressive “hidden” numbers, that’s a huge advantage (especially in a cap system).

Simply rejecting new stats out of hand because they’re not perfect or because they don’t tell the whole story or because you don’t understand them doesn’t cut it. Neither does going to behindthenet, looking at list of Corsi leaders, and saying “Hahaha Jake Muzzin had the best Corsi of any defenseman, the eggheads must think he’s the best defenseman in the NHL!” which is sort of Simmons’ MO.

Try to have some respect for the field. There is a large community of people who have been doing this for many years now, developing these insights and testing them to see how they perform in the real world. It’s not like some guy decided one day that Corsi was important because “sample size” and everybody just spontaneously agreed.

You don’t have to understand advanced stats to enjoy or have an opinion about the game, but it drives me nuts when people assert that because they personally don’t make any effort to learn about advanced stats, or because they “feel” like Corsi (or whatever) “doesn’t make any sense,” then advanced stats must not have anything useful to add to our understanding of the game.

Posted by Sven22 from Grand Rapids on 05/21/14 at 01:36 PM ET

awould's avatar

Posted by Sven22 from Grand Rapids on 05/21/14 at 01:36 PM ET

Well said.

Posted by awould on 05/21/14 at 01:47 PM ET

Jaromir Blogger's avatar

I have a question: how many big “stats guys” (I don’t know how to define it, so I’ll just leave it open to interpretation) out there have actually played hockey?

This isn’t a leading question and I don’t know what to expect, but I’m curious as to what the answer(s) will be.

Posted by Jaromir Blogger on 05/21/14 at 02:16 PM ET

WingsFanInBeanLand's avatar

I bet this is like porn to TPSH.

Posted by WingsFanInBeanLand from where free agents no longer dare. on 05/21/14 at 02:26 PM ET

J.J. from Kansas's avatar

Posted by jaromirblogger on 05/21/14 at 02:16 PM ET

I believe most of the PPP crew does and Tyler Dellow does.

Gabriel Desjardins, who has actually been employed by NHL teams either did or still does.

Justin Bourne touts the usefulness of advanced stats and he played as high as the AHL level.

I’d say it’s probably about half of them, but that’s based simply on my own memories of interacting with them.

Posted by J.J. from Kansas on 05/21/14 at 03:07 PM ET

Avatar

There are seriously people who think Semyon Varlamov is truly a .927 sv% goalie who will continue to do that forever.

those people are called avalanche fans. for the rest of us, we know he’s nothing more than an average goalie who had an outstanding year. the Avs could easily miss the playoffs next year. in fact I expect they will. it’s kind of like Steve Mason did when he was a rookie carrying his team to the playoffs when the bj’s sucked or that Washington goalie who got them to the Cup Finals 1 year and was out of the league the next. this is why hockey advanced statistics don’t tell us much of a story. we knew those teams were frauds, similar to the habs this year. these stats add to the narrative not define it

Posted by jkm2011 on 05/21/14 at 11:45 PM ET

Nathan's avatar

95% of the things folks accuse the “advanced metrics” crowd of doing are projected onto them, or complete strawmen. None of the prominent voices in the hockey advanced metrics community EVER say that Corsi, or zone starts, or whatever stat, are the end-all, be-all. The are exploring new ground and thus focusing on that ground. But folks are putting words in their mouths when they claim that these guys “don’t use the eyeball test,” etc. etc.

Dellow makes this point pretty clearly—the playoffs are a small sample and of course a team can play well over that sample that didn’t play as well the 82 games prior. It can be due to a goalie, a scoring outburst, whatever. The games are fun to watch and nobody is taking away from that. The point is to say, when you start making judgments on the long-term quality of a player or team based on a 20ish game sample, you’re setting yourself up to be wrong. Which is fine, but how is this a shortcoming of the research these metrics guys are doing?

Posted by Nathan from the scoresheet! on 05/22/14 at 09:04 AM ET

Add a Comment

Please limit embedded image or media size to 575 pixels wide.

Add your own avatar by joining Kukla's Korner, or logging in and uploading one in your member control panel.

Captchas bug you? Join KK or log in and you won't have to bother.

Smileys

Notify me of follow-up comments?

Feed

Most Recent Blog Posts

About Kukla's Korner Hockey

Paul Kukla founded Kukla’s Korner in 2005 and the site has since become the must-read site on the ‘net for all the latest happenings around the NHL.

From breaking news to in-depth stories around the league, KK Hockey is updated with fresh stories all day long and will bring you the latest news as quickly as possible.

Email Paul anytime at pk@kuklaskorner.com

 

image

image