Wednesday, November 15, 2006

 

Fight Club, cont'd.

Wow. A few people wisely said, upon getting word of the Fighting Leads To Winning study, that despite the absurd-on-their-face conclusions, they'd reserve judgement until they saw the actual paper. Improbably, the paper itself appears to be much dumber than preliminary indications. As Vic notes in the comments below, "This stuff is so bad I don't know where to start."

A laundry list, much of which is from Jeff at Sisu and The Forechecker:

1) The fact that a huge majority of fighting majors are coincidental is not addressed (i.e. as Jeff notes, among others, "...how do you distinguish which team is 'sparked?' If the fighting majors are coincidental, shouldn't both teams receive the same benefit?")

2) For some reason, Points is used as the measure of success while Goals Against is used as the measure of failure. (Forechecker notes dryly, "...one would think that Goals For would be the natural complement to Goals Allowed...")

3) Elementary statistical errors, compounded by a lack of common sense. Jeff (my boldface):
Right away, one glaring mistake stands out. To get useful results out of a regression analysis, you have to have independent variables (PIM, ESG, MAJORS etc.) that correlate highly with your dependent variable (PTS) but correlate minimally with each other. [...] Your independent variables should not depend on one another. That's why they're called independent. [...] Given that, how anyone could describe face-offs won and face-offs lost as separate, independent variables is beyond me. The authors of this paper do just that, and conclude that winning face-offs has a greater absolute effect than losing them.

4) Aaack... here's Forechecker with a bunch:
As wins and win percentage have often been employed in production models dealing with professional sports, the NHL does not base their rankings on either of these measures. In contrast, rankings in the NHL are determined by Team Points, where each team receives two points for a win and one point for a tie after regulation.

Obviously the authors don't understand that Team Points are equivalent to Win Percentage X Games Played.

5) Mostly a criticism of the reportage, but a critical distinction:
First, I believe they mistate the numbers in the paper, which refers to the Number of Majors, not the number of penalty minutes served. The coefficient for Majors to Team Points is indeed 0.07587 in this study. What the National Post overlooks is that if you're going to take a Major, you also need to account for the 5 PIM, which has a coefficient of -0.01087, so the net result (even using the paper's shoddy analysis) would be 0.02152, much less than was cited.

6) Conclusions not indicated by the data:
As for the impact on opponent scoring (0.24 Goals Against reduction per Major), I just don't see that in here. The only table included regarding Goals Against uses logarithmic data, so perhaps I'm missing something.

7) Conclusions not supported by the data again (my boldface, as I can't believe the conclusions that the authors & the Post drew considering):
As MAJORS were found to have a positive effect on PTS and a negative effect on GA, this implies that Major Penalties (more specifically fighting) do in fact aid a team’s success.

First of all, there is nothing in the study that suggests that fighting majors have any different effect than other major penalties. The authors are making an unwise leap here.

8) Conclusions not supported by the data again:
Even though fighting results in a penalty, it is shown to be able to jump start a team into action and elicit better play. For example, if a team is not playing well, a player might start a fight with the opposing team in order to get the momentum of the game back on his side.

Again, there is no evidence in the study to suggest this. They certainly don't look at event sequencing, to see if teams score goals after taking major penalties, or instead take those penalties once they have a safe lead.

I realize I've quoted pretty heavily from the Forechecker's site here, but here's his conclusion as well, because it is note-perfect and needs to be read as widely as possible:
Papers like this only serve to cast a disparaging light on the work of hockey statisticians in general, by publicizing absurd claims which fly in the face of both common sense and critical judgement. I'm an economist by training, but I wouldn't jump to use the economist's toolbox in this field before I truly felt confident that the source material is valid, and that the results would be useful. This study fails in both those areas.

Word.

POSTSCRIPT, an hour later: there's actually a bigger lesson here. Most of us will be astonished -- both at the conclusions the authors draw, and the credulous manner in which it was reported -- because we have a decent understanding of the underlying subject matter. But this kind of thing happens all the time, and it's something to keep in mind next time you're watching your local news reporting on the latest (say) medical study -- especially if it confirms your biases.

Friend of the Battle Colby Cosh has built half a career on pointing out ludicrous flaws in academic studies (here's one on environmental cancer) that went otherwise unchallenged. This dude is an anti-smoking activist who has nonetheless taken it upon himself to dispel myths that are promoted by other activists and reported dutifully by the media. (Example: "In still other words, the claim [that non-smokers exposed to secondhand smoke can inhale up to seven times(!) the amount of nicotine inhaled by a smoker] is mathematically impossible."). Caveat lector, folks.

Comments:

OK, I haven't read the article as it doesn't really seem to warrant my time. The title itself makes absolutely no sense, and it's full of holes as you and others have demonstrated.

I still say the biggest problem with their entire premise is that both teams are involved in a fight. Strangely enough, after a fight, one team wins, and one team loses. How they've separated the two game results from the two fighters is beyond me.

- Rod
 


Thanks for the linkage, and keeping this story up front where it needs to be. As you note, this sort of reporting ("a new study today proclaims that...") is a huge problem generally. If that study was simply someone's blog post, I would have been much more charitable in my criticism, but since the National Post picked it up and reported it seriously, I felt a higher level of derision was needed.

I can't believe that the professor actually said to the Post, "The bottom line is, whether you measure it in terms of wins or if you measure it in terms of goals allowed, your major penalties help your team." He's simply throwing caution to the wind and saying the correlation in the numbers translates to cause-and-effect relationships. Without knowing the guy, I'd suspect he doesn't know squat about hockey and is simply looking for some publicity and/or grant money.
 


Without knowing the guy, I'd suspect he doesn't know squat about hockey and is simply looking for some publicity and/or grant money.

One of the authors is Stacey Brook, who was involved with the Sports Economistguys on The Wages of Wins. I'm not an economist, but my understanding was that they are fairly reputable guys. I was therefore a little taken aback to see her name attached to what so many people are saying is a poor document.

As for Fenn, here is his sports economics resume.
 


... What the National Post overlooks is that if you're going to take a Major, you also need to account for the 5 PIM, which has a coefficient of -0.01087, so the net result (even using the paper's shoddy analysis) would be 0.02152, much less than was cited.

Good spot, also a big chunk of misconduct penalties come along with a major, and presumably misconduct PIMs are being included there. In any case, even without considering the misconduct PIM effect ... with your math here it would take about FIFTY extra majors in a season to garner ONE extra point in the standings. And surely even that is far too generous.

Granted there is so much that is wrong here, it's tough to settle on anything in particular.

In the points equation they've started by giving everyone 80 points and nicking them for some shit and crediting them a bit for others. Start throwing in all sorts of mad shit as variables (combined weight of alternate captains, number of vowels in Coach's surname, etc.) and sooner or later you'll be able to make some mad shit seem plausible at first blush.
 


Caveat Lector ...is that Beware of Hannibal Lector? Cuz that seems like good advice. And a piece of more useful information than anything I found in that paper (though he cites another published study that seems to find a correlation between salary and violence -- ie, goons are paid less than stars. Which seems about as self evident as caveat lector.)
 


Re:"Both teams in a fight"
In regards to the matching penalty "conundrum", I think people are jumping the gun a little. I haven't read the study, so I say this as an out-of-left-field devil's advocate, but couldn't you argue that fighting majors may benefit one team over the other, if you allow that the "emotional lift" effect lasts longer than the one game?
Example:
Team A's goon gets in a fight with Team B's goon. They both get major penalties.

The next night, Team A plays Team C. Team C would love to lay a big hit on Team A's star player, but don't because Team A's goon displayed his toughness the night before. Team A hits Team C with impunity, and win. Team B wins against Team D the same way. After C and D play (say C wins), you get:
Team A - 2 wins - 1 major
Team B - 1-1 - 1 major
Team C - 1-1
Team D - 0-2

This could allow for a correlation with fighting majors and wins. Of course, it sounds like the report in question isn't making this claim, so please don't think I'm defending a shoddy report.
 


One of the authors is Stacey Brook, who was involved with the Sports Economistguys on The Wages of Wins. I'm not an economist, but my understanding was that they are fairly reputable guys. I was therefore a little taken aback to see her name attached to what so many people are saying is a poor document.

She IS a girl Andy.
 


Huh?

I know she is a girl. Did I ever say she wasn't?
 


Considering what you've outlined here, if anything, the paper seems to be accidentally impling that dirty play, i.e. non-coincidental majors, is what really wins hockey games, all other things being equal. (Which they're not by any stretch of the imagination, as demonstrated here, but I digress.)
 


No! Crap, Colorado College should have rung a bell.

Also, re: "impressed" -- it was more the latter than the former, BUT, the big deal was always (C) that an NHLPA member wrote a paper that indirectly advocated a hard salary cap.
 

Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?