Monday, July 16, 2007
The 6.4 Million Dollar Question
Lowetide brings up something that's been nagging at me for just about the lifetime of this site:
100% true, of course. He transcribes "analysis" of every NHL and WHA coach from 1978 and asks three questions:
The answer to the first is, approximately, nothing. The answer to the third is definitely; there are blogs, forums, pub tables, and many many more venues that have seen hours and hours of time, and mental horsepower, devoted to discussing this question. Is the coach doing just what he needs to be doing, or is his head up his ass? Pick any pro sports team and you can find fans who will passionately argue the affirmative, and fans who will passionately argue the negative.
The second question is the tough one, and I think my answer is, I'm not so sure that there is.
The optimist in me says surely we can get somewhere with this. There is so much more information available now than there was 30 years ago, and we've come a very long way developing some more sophisticated statistical bases for evaluating players (I say bases because I don't want to claim they're definitive).
The pessimist in me says dream on. As I say, this broad question is not limited to hockey at all. Baseball has the best statistics, and is the most suitable for statistical analysis, but there is still very little consensus as to who the best managers are, or what makes a good manager. Or how big a difference good manager vs. bad manager makes on results.
The very fact that there is essentially no accepted objective ways to evaluate coaching -- despite that there are so many people interested in (arguing about) it -- leads me to believe that it'll never happen. (And if you're tempted to answer "Scoreboard" here, my follow-up questions would be (1) Which one? and (2) Remind me how you determined how big the coach's impact was on the scoreboard again?)
Surely the most optimistic-yet-realistic answer to Lowetide's second question is "Yes, but we really don't know what those questions are." My only bright idea here is to use some plurisapience. I wonder: would a broad survey of attentive hockey fans, on the topic of what makes good coaching, produce a more accurate picture of reality than the alternatives? (One alternative being the considered opinion of a smart guy who's been following hockey for a long time, like Lowetide or Duhatschek?)
My thought would be to construct a multiple-choice poll-type-thingy on a broad range of the elements of coaching. It would mainly serve, if I guess correctly, to identify maybe a couple of areas where there actually is a broad consensus. Something like:
1) Coach A's .500 team is excellent at even strength but has poor special teams. Coach B's .500 team has excellent special teams but is poor at even strength. Who is the better coach?
2) Coach A and B both have teams that finished with 100 points and scored 30 more goals than they allowed. Broken down by period though, Coach A's team had all their +30GD in the 1st period (+30/0/0), and Coach B's team had all their +30GD in the 3rd (0/0/+30). Is Coach A better prepared, or does he have a great team which he undercuts by failing to make good adjustment? Is Coach B superb at making in-game adjustments, or does he prepare his team poorly? (Or is this entirely a physical fitness issue?) Who is the better coach?
3) CoachP's A's team is great at home, terrible on the road. Coach B's team is poor at home but great on the road. Who is the better coach?
4) Essay question: Name one way in which you would evaluate the performance of Coach X that is distinct (or distinguishable) from how you would evaluate the performance of X's team.
- Put another way, if there was a good coach who was running a team that was performing below expectations, how would you know? How about a bad coach whose team was nevertheless exceeding preseason expectations? These things have happened -- is there evidence you can point to?
etc. To me, to ask these questions is to realize that they're probably unanswerable. Then again, maybe I'm just looking at the question too broadly and getting needlessly discouraged. If we came up with a hockey equivalent for "putting a guy with a sub-.300 OBP in your leadoff spot costs your team runs and thus games", it would constitute a major advance. I'll be fascinated to see where Lowetide goes from here.
There's a lot we don't know about coaches, and it isn't getting any better.
100% true, of course. He transcribes "analysis" of every NHL and WHA coach from 1978 and asks three questions:
...29 years later how much more do we know about NHL coaches and what they do to help their teams win? Aren't there a group of questions we can ask about them (starting this winter) in order to improve our knowledge of them? Is anyone else interested in this stuff?
The answer to the first is, approximately, nothing. The answer to the third is definitely; there are blogs, forums, pub tables, and many many more venues that have seen hours and hours of time, and mental horsepower, devoted to discussing this question. Is the coach doing just what he needs to be doing, or is his head up his ass? Pick any pro sports team and you can find fans who will passionately argue the affirmative, and fans who will passionately argue the negative.
The second question is the tough one, and I think my answer is, I'm not so sure that there is.
The optimist in me says surely we can get somewhere with this. There is so much more information available now than there was 30 years ago, and we've come a very long way developing some more sophisticated statistical bases for evaluating players (I say bases because I don't want to claim they're definitive).
The pessimist in me says dream on. As I say, this broad question is not limited to hockey at all. Baseball has the best statistics, and is the most suitable for statistical analysis, but there is still very little consensus as to who the best managers are, or what makes a good manager. Or how big a difference good manager vs. bad manager makes on results.
The very fact that there is essentially no accepted objective ways to evaluate coaching -- despite that there are so many people interested in (arguing about) it -- leads me to believe that it'll never happen. (And if you're tempted to answer "Scoreboard" here, my follow-up questions would be (1) Which one? and (2) Remind me how you determined how big the coach's impact was on the scoreboard again?)
Surely the most optimistic-yet-realistic answer to Lowetide's second question is "Yes, but we really don't know what those questions are." My only bright idea here is to use some plurisapience. I wonder: would a broad survey of attentive hockey fans, on the topic of what makes good coaching, produce a more accurate picture of reality than the alternatives? (One alternative being the considered opinion of a smart guy who's been following hockey for a long time, like Lowetide or Duhatschek?)
My thought would be to construct a multiple-choice poll-type-thingy on a broad range of the elements of coaching. It would mainly serve, if I guess correctly, to identify maybe a couple of areas where there actually is a broad consensus. Something like:
1) Coach A's .500 team is excellent at even strength but has poor special teams. Coach B's .500 team has excellent special teams but is poor at even strength. Who is the better coach?
2) Coach A and B both have teams that finished with 100 points and scored 30 more goals than they allowed. Broken down by period though, Coach A's team had all their +30GD in the 1st period (+30/0/0), and Coach B's team had all their +30GD in the 3rd (0/0/+30). Is Coach A better prepared, or does he have a great team which he undercuts by failing to make good adjustment? Is Coach B superb at making in-game adjustments, or does he prepare his team poorly? (Or is this entirely a physical fitness issue?) Who is the better coach?
3) Coach
4) Essay question: Name one way in which you would evaluate the performance of Coach X that is distinct (or distinguishable) from how you would evaluate the performance of X's team.
- Put another way, if there was a good coach who was running a team that was performing below expectations, how would you know? How about a bad coach whose team was nevertheless exceeding preseason expectations? These things have happened -- is there evidence you can point to?
etc. To me, to ask these questions is to realize that they're probably unanswerable. Then again, maybe I'm just looking at the question too broadly and getting needlessly discouraged. If we came up with a hockey equivalent for "putting a guy with a sub-.300 OBP in your leadoff spot costs your team runs and thus games", it would constitute a major advance. I'll be fascinated to see where Lowetide goes from here.
# posted by Matt @ 2:33 pm 
Comments:
Even before you get to evaluating coaches, there's the question of their tendencies. At least in baseball, there are no people actively cataloguing managerial tendencies - this year's BPro featured little rundowns on all the players. I'd think that something similar could be done in hockey, at the very least with whether they lean on their top ends or not.
Point taken. I think plenty of people would be interested in that information if it was logged and assembled.
I don't think I personally would be pushing for such a thing -- I'm impatient and lack imagination, so the idea of compiling all sorts of information just for the sake of having it (without really knowing what to do with it, or even if it'd be useful) doesn't really intrigue me. But who knows what comes out in the wash, I suppose... maybe someone figures out a way to quantify (to some degree) how intensely a coach line-matches their top personnel, home and road. This seems plausible, and if you're lucky, some patterns jump out at you. I'd still want to start the whole process with some kind of hypothesis to test, though.
I've never really given it much thought, but as a first impression I'd say that a coach's job is to maximize the value of the players at his disposal, so the best way to measure coaching is to measure a team's expected results based on its players, then find the coaches whose teams consistently exceed those expectations. Then maybe you could try to find the traits that the "best" coaches have in common.
The major flaw with that plan is that I have no idea if it's possible to measure a team's expected results.
5) Compare and Contrast an ideal NHL coach with the ancient coaches of Mesopotamia. Use examples from our discussions in class or examples from the course text. Bonus points: If Scotty Bowman could travel back in time and speak Sumerian would the 'power forward' have developed thousands of years previously?
Great analysis. It almost begs the same questions when looking at organizational effectiveness since the coach is essentially a front-line manager. I happen to be a front-line manager. Depending on the personel, you adjust your style to that individual. Some employees need to be told everyday that they are doing an amazing job, while others need to be ridden, some like more money while other want more recognition, etc etc....
Without knowing more about the players personalities and how the coach is able to manage these differing personalities, I don't know if you'll ever be able to effectively evaluate coaches from the outside looking in. It all depends on the expectations that "upper management" has. Regardless of how realistic that might be.
There are so many variables here, it's hard to know where to begin.
What I'd like to do is take someone who is a phenomenal coach and have him grade others' work. A lot of what goes on has to do with systems and line matching more than who plays how many minutes... in my mind, it's less about who he uses and more about how a coach deploys his players that I would be looking to evaluate.
I'd love to get more into coaching theory and take a couple of courses. It all still seems pretty primitive in hockey; a guy like Hitchcock is one of the "geniuses" and he never played and rose up through the ranks by coaching minor hockey near Edmonton. A few years in Kamloops, and he was off to the NHL.
Is that luck? I doubt it. But what is Hitch doing that so many others fail to do?
What I'd like to do is take someone who is a phenomenal coach and have him grade others' work.
Huh? Isn't the whole idea that we don't know what makes a phenomenal coach?
What I'd like to do is take someone who is a phenomenal coach and have him grade others' work.
Also, asking sports people to evaluate one another has something of a problematic history as far as tangible statistical analysis is concerned.
Scott Bowman was pretty good.
Great stuff Matt but I would say brent's point is an excellent one. Who knows who has no confidence (Raffi Torres) and who is a cokehead? I coached hockey and soccer for years (albeit at a very low level) and every kid was different. Some would go through a brick wall for you without a word - that was just their makeup. Some needed to be pushed, others babied. I think once you're in the pros things aren't much different - obviously more complicated but you have such a collection of individuals ...
Look at Cito Gaston. A lot of Jays fans hated him and thought he was a terrible manager. All he did was write out the lineup card they said.
But a lot of talented teams don't win and Gaston's players themselves respected him. They carried themselves as professionals and they played hard. Just the team's makeup you might say? Well, he has as many World Series rings as Tony LaRussa and LaRussa had some pretty formidable teams over the years. Guy is a genius but he has had teams that folded bigtime.
What about Keenan? Great coach in Philly, Chicago, NY? Suddenly a bad coach elsewhere?
From a statistical (scientific) basis it's virtually impossible to separate a coach from the players.
Secondly, each coach adds value to different aspects of the game (as noted in the post). Is a special teams based coach more value than a coach who focuses on plus-minus related performance.
The biggest problem, however, is determining what is luck and what is coaching. Due to the nature of coaching there are no in game statistics to work with (at this point) so most metrics will look at winning percentage (normalized for player skill). However you need about 400 games (for +/- 5%; with good estimates for player skill over all those 400 games). The simplest method to do this would be to look at winning percentage and salary for each coach.
The second trick is to look at individual player's stats over their careers with different coaches. (You need defensive stats to even the playing field for defense first coaches). Did goals scored go up on average for all players (normalizing age/injury related factors)
You could do a regression of 1000 players and 30+ coaches:
goals scored = Players score + Coaches score
eg. Naslund averaged about 40 goals/season with Crawford and 20 per season without Crawford, which gives Crawford half of Naslund's scoring when Crawford was coaching (note: biased sample).
This is of course assumes you could get the available data into a useful format without too much hassle.
There's only one way to evaluate a coach - his record. Good coaches find ways to win, they change the way the game is played. Successful innovation is probably an important attribute - Nielsen brought in video review and Bowman was the first guy to asign a shadow to a defenceman.
But it is a quagmire of a problem.
If we came up with a hockey equivalent for "putting a guy with a sub-.300 OBP in your leadoff spot costs your team runs and thus games"
See putting Craig Simpson in charge of the Oilers powerplay.
i think the strategy analysis requires an FBI profiler first biograph the guy. then a shrink to document coach-player relationship handling.
then...
then...
aw crap. this is hard for my puny brain.
how do you grade military commanders in action? by how many men killed?
Post a Comment
<< Home
Even before you get to evaluating coaches, there's the question of their tendencies. At least in baseball, there are no people actively cataloguing managerial tendencies - this year's BPro featured little rundowns on all the players. I'd think that something similar could be done in hockey, at the very least with whether they lean on their top ends or not.
# posted by 
: July 16, 2007 3:39 pm
: July 16, 2007 3:39 pm Point taken. I think plenty of people would be interested in that information if it was logged and assembled.
I don't think I personally would be pushing for such a thing -- I'm impatient and lack imagination, so the idea of compiling all sorts of information just for the sake of having it (without really knowing what to do with it, or even if it'd be useful) doesn't really intrigue me. But who knows what comes out in the wash, I suppose... maybe someone figures out a way to quantify (to some degree) how intensely a coach line-matches their top personnel, home and road. This seems plausible, and if you're lucky, some patterns jump out at you. I'd still want to start the whole process with some kind of hypothesis to test, though.
# posted by : July 16, 2007 3:53 pm
: July 16, 2007 3:53 pm I've never really given it much thought, but as a first impression I'd say that a coach's job is to maximize the value of the players at his disposal, so the best way to measure coaching is to measure a team's expected results based on its players, then find the coaches whose teams consistently exceed those expectations. Then maybe you could try to find the traits that the "best" coaches have in common.
The major flaw with that plan is that I have no idea if it's possible to measure a team's expected results.
5) Compare and Contrast an ideal NHL coach with the ancient coaches of Mesopotamia. Use examples from our discussions in class or examples from the course text. Bonus points: If Scotty Bowman could travel back in time and speak Sumerian would the 'power forward' have developed thousands of years previously?
# posted by : July 16, 2007 5:06 pm
: July 16, 2007 5:06 pm Great analysis. It almost begs the same questions when looking at organizational effectiveness since the coach is essentially a front-line manager. I happen to be a front-line manager. Depending on the personel, you adjust your style to that individual. Some employees need to be told everyday that they are doing an amazing job, while others need to be ridden, some like more money while other want more recognition, etc etc....
Without knowing more about the players personalities and how the coach is able to manage these differing personalities, I don't know if you'll ever be able to effectively evaluate coaches from the outside looking in. It all depends on the expectations that "upper management" has. Regardless of how realistic that might be.
# posted by : July 16, 2007 5:28 pm
: July 16, 2007 5:28 pm There are so many variables here, it's hard to know where to begin.
What I'd like to do is take someone who is a phenomenal coach and have him grade others' work. A lot of what goes on has to do with systems and line matching more than who plays how many minutes... in my mind, it's less about who he uses and more about how a coach deploys his players that I would be looking to evaluate.
I'd love to get more into coaching theory and take a couple of courses. It all still seems pretty primitive in hockey; a guy like Hitchcock is one of the "geniuses" and he never played and rose up through the ranks by coaching minor hockey near Edmonton. A few years in Kamloops, and he was off to the NHL.
Is that luck? I doubt it. But what is Hitch doing that so many others fail to do?
What I'd like to do is take someone who is a phenomenal coach and have him grade others' work.
Huh? Isn't the whole idea that we don't know what makes a phenomenal coach?
What I'd like to do is take someone who is a phenomenal coach and have him grade others' work.
Also, asking sports people to evaluate one another has something of a problematic history as far as tangible statistical analysis is concerned.
Scott Bowman was pretty good.
Great stuff Matt but I would say brent's point is an excellent one. Who knows who has no confidence (Raffi Torres) and who is a cokehead? I coached hockey and soccer for years (albeit at a very low level) and every kid was different. Some would go through a brick wall for you without a word - that was just their makeup. Some needed to be pushed, others babied. I think once you're in the pros things aren't much different - obviously more complicated but you have such a collection of individuals ...
Look at Cito Gaston. A lot of Jays fans hated him and thought he was a terrible manager. All he did was write out the lineup card they said.
But a lot of talented teams don't win and Gaston's players themselves respected him. They carried themselves as professionals and they played hard. Just the team's makeup you might say? Well, he has as many World Series rings as Tony LaRussa and LaRussa had some pretty formidable teams over the years. Guy is a genius but he has had teams that folded bigtime.
What about Keenan? Great coach in Philly, Chicago, NY? Suddenly a bad coach elsewhere?
From a statistical (scientific) basis it's virtually impossible to separate a coach from the players.
Secondly, each coach adds value to different aspects of the game (as noted in the post). Is a special teams based coach more value than a coach who focuses on plus-minus related performance.
The biggest problem, however, is determining what is luck and what is coaching. Due to the nature of coaching there are no in game statistics to work with (at this point) so most metrics will look at winning percentage (normalized for player skill). However you need about 400 games (for +/- 5%; with good estimates for player skill over all those 400 games). The simplest method to do this would be to look at winning percentage and salary for each coach.
The second trick is to look at individual player's stats over their careers with different coaches. (You need defensive stats to even the playing field for defense first coaches). Did goals scored go up on average for all players (normalizing age/injury related factors)
You could do a regression of 1000 players and 30+ coaches:
goals scored = Players score + Coaches score
eg. Naslund averaged about 40 goals/season with Crawford and 20 per season without Crawford, which gives Crawford half of Naslund's scoring when Crawford was coaching (note: biased sample).
This is of course assumes you could get the available data into a useful format without too much hassle.
There's only one way to evaluate a coach - his record. Good coaches find ways to win, they change the way the game is played. Successful innovation is probably an important attribute - Nielsen brought in video review and Bowman was the first guy to asign a shadow to a defenceman.
But it is a quagmire of a problem.
If we came up with a hockey equivalent for "putting a guy with a sub-.300 OBP in your leadoff spot costs your team runs and thus games"
See putting Craig Simpson in charge of the Oilers powerplay.
i think the strategy analysis requires an FBI profiler first biograph the guy. then a shrink to document coach-player relationship handling.
then...
then...
aw crap. this is hard for my puny brain.
how do you grade military commanders in action? by how many men killed?
# posted by : July 17, 2007 6:47 pm
: July 17, 2007 6:47 pm Post a Comment
<< Home
