Scouting the scouts

[random-d]

Funny cause I'm reading a book call Noise that explain why human-being are making errors when manking a judgment and how to bypass them. In fact, where there is a human judgment, there is always noise and bias that provoke error. This clearly apply to the context of hockey scouting; noise and bias can significantly diminish the accuracy of judgment. Noise, stemming from fluctuations in player performance or scout evaluations, can lead to overemphasizing individual game outcomes or streaks, making it challenging to gauge a player's true long-term potential. Bias can arise from subjective preferences, regional loyalties, or stereotypical assumptions about players, distorting the assessment process. For instance, if a scout has a bias towards selecting players from their hometown or prefers certain playing styles, it may lead to overlooking promising talent or making unfair judgments. It is proven that even a simple modelization results in better outcomes than human judgement. So, to identify the better public scout, I would look for those that use data-driven models that can reduce noise by considering comprehensive player statistics and mitigate bias by evaluating players based on standardized criteria, ultimately enhancing the ability to identify and develop top prospects more fairly and accurately.

But you also have to keep in mind that models have a difficult time coming to accurate conclusions when dealing with complex systems that have a myriad of variables that cannot be adequately accounted for. There is no data driven model than can account for the coach, the quality of competition, the personal life of the player, day to day variations in perfomance and circumstances etc. Data based models can provide useful information but they're almost equally as fallible as human assessments because they are unable to account for the very things that they intend to compensate for.

[marc1pol2]

In fact, you can modelize almost anything when you got the information... for a basic example you could rank on a scale of 1 to 10 the quality of competition in a league, the habits of the players, etc...

[random-d]

I would even go so far as to say that if you suggest you're only going to decide based on quantifiable metrics you are just introducing another bias into a system that has non-quantifiable metrics.

[marc1pol2]

I'm not saying that it would result in perfect "prediction", such a thing doesnt exist and a model would indeed have bias. But just by eliminating noise (random or unpredictable variations, errors, or inconsistencies in the decision-making process that can lead to different outcomes or assessments when the same decision is made multiple times, even under the same conditions.), it would lead to better prediction. It's not my opinion. it's a fact based on multiples experiences in multiple field of study (justice, hiring process, economy, politics, social science, etc. etc.).

[marc1pol2]

I would even go so far as to say that if you suggest you're only going to decide based on quantifiable metrics you are just introducing another bias into a system that has non-quantifiable metrics.

As you look interested and knowledgeable on the question, I suggest you read Noise wirtten by Nobel prized in economics Daniel Nahneman

[random-d]

I'm not saying that it would result in perfect "prediction", such a thing doesnt exist and a model would indeed have bias. But just by eliminating noise (random or unpredictable variations, errors, or inconsistencies in the decision-making process that can lead to different outcomes or assessments when the same decision is made multiple times, even under the same conditions.), it would lead to better prediction. It's not my opinion. it's a fact based on multiples experiences in multiple field of study (justice, hiring process, economy, politics, social science, etc. etc.).

Give me an example of how statistical modeling would produce a predictable outcome and I'll give you an example of how that certainty is greatly diminished when dealing with complex data sets that contain non-quantifiable data. I'm not going to tell you, because I'm sure you know, that statistical outcomes don't typically deal in certainties. They deal in probabilities, and those probabilities are diminished when there are variables that cannot be accounted for with statistics.

[marc1pol2]

Give me an example of how statistical modeling would produce a predictable outcome and I'll give you an example of how that certainty is greatly diminished when dealing with complex data sets that contain non-quantifiable data. I'm not going to tell you, because I'm sure you know, that statistical outcomes don't typically deal in certainties. They deal in probabilities, and those probabilities are diminished when there are variables that cannot be accounted for with statistics.

I'm just saying that even in a complex environement, it is proven that a simple model leads to equally or better prediction than human just by eliminating the noises. Its not perfect, it would never be but it is for a reason that now, every NHL team got a statistical department and that they use stats more and more in the scouting decision.

[Eskimo Brother]

Appreciate both sides of that guys, and that's exactly what I'm trying to measure out here. Hoping to include a lot of the analytical guys like Scouch, Crossbarr, Bader, Thibaud, etc, plus the glut of public scouts that do it based on viewings and more personal analysis/bias.

Not sure how long this is going to take me or how much time I will have to put into this, but I'm hoping that I can get it sorted out in time for it to be actionable for getting our 2024 draft lists organized.

[random-d]

I will say one last thing to make it clear: Statistical modeling is value-adding, however it is not (on it's own) determinate.

One major shortcoming of statistical modeling is that very few typically report how or how often it fails, either by incorrectly identifying something or not identifying something at all.

[metaldude26]

I will say one last thing to make it clear: Statistical modeling is value-adding, however it is not (on it's own) determinate.

One major shortcoming of statistical modeling is that very few typically report how or how often it fails, either by incorrectly identifying something or not identifying something at all.

It's one thing I really like about Byron Bader's HockeyProspecting model. He shows that guys who fit this profile make it X percentage of the time. It is very helpful in just contextualizing your draft decisions in terms of probabilities.

[random-d]

It's one thing I really like about Byron Bader's HockeyProspecting model. He shows that guys who fit this profile make it X percentage of the time. It is very helpful in just contextualizing your draft decisions in terms of probabilities.

I think one of the biggest challenges with predicting prospects is that you're trying to do so (typically) 2-3 years in advance at minimum. Inevitably there are alot of values that could change during that time. For an easy to understand comparison for that type of modeling I would point people to the event modeling done by the NOAA and the NWS. The best accuracy they can typically achieve within 72 hours of a possible event is around 15-30% and that can drop to 0% within 12 hours. The confidence increases with proximity to expected occurrence time but it still never approaches certainty until it gets within 12 hours or less of the expected time. This is a great example of what you can expect from statistical modeling. It points in the direction of where you should look and it can rule out where you shouldn't look, but it does not tell you that it will definitely rain in the place where you're standing.

[chuckcouples]

Statistical analysis is a great value add to all worlds is something I completely agree with. Saying that it's better in hockey scouting than watching players play and interacting with them and their people, is false. There are way too many undeterminable "stats" to create a model that would beat most nhl teams scouting department. If something like that could be done, it would be as nhl teams would pay millions of dollars for a statistical system that would save them the extraordinary costs of having their scouts travel to and watch as many games as possible each year.

I am pretty sure all teams use some form of statistical analysis as part of their process but there is still way more information to be trained by watching the players' games in person. Scouts typically go to a game to watch 2 - 4 players per game. They watch their entire shifts and even what they do before and after they get on the ice for their shifts. There is lots to be learned from thar type of stuff that sound never be seen by data or by watching game film.

[Axeman33]

Statistical analysis is a great value add to all worlds is something I completely agree with. Saying that it's better in hockey scouting than watching players play and interacting with them and their people, is false. There are way too many undeterminable "stats" to create a model that would beat most nhl teams scouting department. If something like that could be done, it would be as nhl teams would pay millions of dollars for a statistical system that would save them the extraordinary costs of having their scouts travel to and watch as many games as possible each year.

I am pretty sure all teams use some form of statistical analysis as part of their process but there is still way more information to be trained by watching the players' games in person. Scouts typically go to a game to watch 2 - 4 players per game. They watch their entire shifts and even what they do before and after they get on the ice for their shifts. There is lots to be learned from thar type of stuff that sound never be seen by data or by watching game film.

Listening to that podcast I mentioned earlier, when they are talking to an actual NHL scout, really explains a lot of what they do in person at the games they attend. It's more than just watching the games.

[metaldude26]

I think one of the biggest challenges with predicting prospects is that you're trying to do so (typically) 2-3 years in advance at minimum. Inevitably there are alot of values that could change during that time. For an easy to understand comparison for that type of modeling I would point people to the event modeling done by the NOAA and the NWS. The best accuracy they can typically achieve within 72 hours of a possible event is around 15-30% and that can drop to 0% within 12 hours. The confidence increases with proximity to expected occurrence time but it still never approaches certainty until it gets within 12 hours or less of the expected time. This is a great example of what you can expect from statistical modeling. It points in the direction of where you should look and it can rule out where you shouldn't look, but it does not tell you that it will definitely rain in the place where you're standing.

I am very accustomed to the uncertainties with weather given my father was a weatherman. It's also heavily tied in to the work I do in forest fire fighting. It's crazy the certainty that people will talk about certain weather patterns and such. My favourite is people guessing if it's going to be a busy fire season in like March. Like, how in the hell can anyone predict this? It's good perspective to keep in mind when forecasting prospects. Outside of the outliers at the top of the draft most prospects are at best a 50/50 bet and the majority with much worse odds with the data you've accumulated to that point in their careers.

[Axeman33]

I am very accustomed to the uncertainties with weather given my father was a weatherman. It's also heavily tied in to the work I do in forest fire fighting. It's crazy the certainty that people will talk about certain weather patterns and such. My favourite is people guessing if it's going to be a busy fire season in like March. Like, how in the hell can anyone predict this? It's good perspective to keep in mind when forecasting prospects. Outside of the outliers at the top of the draft most prospects are at best a 50/50 bet and the majority with much worse odds with the data you've accumulated to that point in their careers.

Are you saying you don't purchase the Farmers Almanac for it's year long forecasting of the weather?