Best guide from the 2022/23 season

[peeknpoke]

Hello poolers,

Want to know which hockey pool list of projections was best for a one-year, straight points-only draft from the 2022/23 season?

I did, so over the course of this summer I have executed and evaluated over 78.3 million hockey pool draft scenarios using the 48 projection lists which were collected last season.

You can find all my videos breaking down the tests and results here Hockey Pool Aid Videos on YouTube

Over-all there were five lists which made it to the top levels as finalists in both the 8 owner and 9 owner drafts. They were:
Steve's Cheat Sheet - Balanced and successful in many draft lengths
The Hockey News - very strong in the shorter drafts
Dom Luszczysyn - Performed better in longer drafts
Apples & Ginos - Was a presence in mid to long range drafts but couldn't take the top spot often enough
Pool-Hockey - A finalist which shows that they could be successful against lesser opponents

Another five lists were at the top level as finalists in one but not both sets of owner drafts. They were:
CanadianSandGoggles - did an amazing job in the 8-owner battles but not as fortunate in the 9-owner battles
Le Droit / Le Soleil - If only they produced a longer list we would be able to better judge their strength in long drafts
Crome Aggregate Projections - The 9-owner Champion but a little less consistent in the 8-owner fights
Left Wing Lock - Can flirt with the top spot in the mid length drafts
Ultimate Pool Preview - Challenges in the short to mid length drafts

The maximum draft lengths were limited by the smallest list which belonged to Le Droit / Le Soleil. If Le Droit / Le Soleil were not so successful around the 300 drafted player mark the maximum draft lengths could have been in the high-400s or low-500s range, thus allowing lists by Evolving-Hockey, Dom Luszczysyn, and Steve's Cheat Sheet to showcase themselves a little bit more.

While there were some pretty good challengers, the guide with the most consistency and success across all the formats which we tested was Steve's Cheat Sheet!!! Congrats to them!!!

I have already started to collect the projections lists for the 2023/24 season and I look forward to putting them through even more rigorous testing next summer.

Good luck in your pools this season!!!

[random-d]

Why 8 and 9 owner? The most common league sizes are 10-14. On the surface that says I shouldn't even consider this information.

I also fail to find value in something that fails 10-20 times more than it succeeds as the 8th and 9th place guides did. Likewise the number of guides in the "finals" that finished last more often that they finished 1st suggests to me that they're not much better than a coin flip without knowing the distribution of placements in between those results.

It would be more valuable to know how consistently they finished in what is typically the money positions (1-3) rather than just how often they won. Even better still how they perform when their finishing positions are averaged. I would prefer the guide with a higher average performance over the one that loses more times than it wins.

[ross10019]

Was the Dobber Guide part of your review?

I find it hard to believe it would not make these lists.

[random-d]

Was the Dobber Guide part of your review?

I find it hard to believe it would not make these lists.

It's listed but it's not clear whether its the actual fantasy guide or not because they're also listed separately as the "Dobber top 300", which presumably is the top 300 keeper league skaters list.

I also didn't see anything about what the standardized scoring categories were and whether or not the guides were customized for those categories.

Altogether this is an incredibly flawed way to determine what the best guide is. It's an interesting exercise, but I wouldn't call it determinative by any stretch.

[peeknpoke]

Yes, the guide and the monthly top 300 lists were evaluated.

Dobber's guide was in the second tier. I think it is because most aggregator's include his projections.

[peeknpoke]

I clearly state in my videos that it is only on one category, PTS.

If you can evaluate groups of 10 guides from a total of 48 guides where draft order is important and do so at 6 different draft lengths then you would have evaluated
142,409,699,060,428,800 possible scenarios. that's 142 quadrillion! I'll happily read your analysis.

Oh and for 14 guides, I think it is over 250 sextillion scenarios.

So, thank you and if you fail to find value in evaluating over 78 million scenarios then I can' t help you.

[random-d]

I clearly state in my videos that it is only on one category, PTS.

If you can evaluate groups of 10 guides from a total of 48 guides where draft order is important and do so at 6 different draft lengths then you would have evaluated
142,409,699,060,428,800 possible scenarios. that's 142 quadrillion! I'll happily read your analysis.

Oh and for 14 guides, I think it is over 250 sextillion scenarios.

So, thank you and if you fail to find value in evaluating over 78 million scenarios then I can' t help you.

Lol, do you honestly think that repeating a flawed experiment to the nth power removes the flaws? The numbers speak for themselves. Your top 9 contains 2 guides that failed 10-20 more times than they succeeded. While you've successfully demonstrated that you can collect data for a large number of iterations you have not successfully demonstrated that the data has any qualitative value. Maybe you can, but thus far, you have not.

[peeknpoke]

You keep mentioning a flaw but what you are really describing that your expectation is not being met.

I run these tests with an open mind as to what the outcome should or could be.

You seem to think that after the 48 guides are whittled down to the final 8 or 9, that everything will be close in the final set of battles. That every guide will have approximately a 12.5% winning rate. That is not the case.
Even if you ran the tests using only one guide for every owner, not all owners will win equally.

If you stop fixating on the number of losses by some of the finalists then maybe you could appreciate what was done here. The fact that they got to the finals is proof that they were able to beat most of the other guides, most of the time.

As far as the points of error or flaws with this process I freely admit that I am limited to 8 and 9 owner drafts. I also try to mitigate that by running single tests of longer drafts in terms of round and in terms of number of owners. I also run a box-pool evaluation too.

[random-d]

You keep mentioning a flaw but what you are really describing that your expectation is not being met.

I run these tests with an open mind as to what the outcome should or could be.

You seem to think that after the 48 guides are whittled down to the final 8 or 9, that everything will be close in the final set of battles. That every guide will have approximately a 12.5% winning rate. That is not the case.
Even if you ran the tests using only one guide for every owner, not all owners will win equally.

If you stop fixating on the number of losses by some of the finalists then maybe you could appreciate what was done here. The fact that they got to the finals is proof that they were able to beat most of the other guides, most of the time.

As far as the points of error or flaws with this process I freely admit that I am limited to 8 and 9 owner drafts. I also try to mitigate that by running single tests of longer drafts in terms of round and in terms of number of owners. I also run a box-pool evaluation too.

Do you have a negative control? Maybe a randomized player list (several would be ideal). What about a positive control? Aggregate of all lists? I would also remove any consensus picks since picking 1st seemed to skew results. The value of a guide is it's ability to provide value beyond the "knowns" (i.e. the generally agreed upon best values).

[By "negative control" I mean something that controls for draft position variation and compares it to how different it is from random.]

Maybe I'm not sure what you're attempting to provide here. Is it intended to be a qualitative answer to which guide is best or it intended to be a fun exercise with some numbers? It is certainly not the former, but if that wasn't the purpose than I feel fine with it.

[ross10019]

Yes, the guide and the monthly top 300 lists were evaluated.

Dobber's guide was in the second tier. I think it is because most aggregator's include his projections.

If Dobber is included by others / aggregators isn’t that super important to point out, as it contributes to the purported value of those guides and yet they finish ahead of him? I’m confused.

[random-d]

It sure would be useful to provide a video or explanation of how the simulations are run. Does every guide get an equal number of iterations at each draft position or are the draft positions randomly assigned for each one. That would have a huge impact on the results. How does the simulation account for player injuries/absences? It wouldn't be representative of an actual outcome if it just counted an injured player as a "zero" because in reality that roster spot would be vacated and filled by another player. So you would have to remove any players whose seasons were anomalous due to prolonged absence.

[peeknpoke]

This is stated in most of my videos. Battle parameters for 8-Owner, 9-Owner and Draft-a-thon testing:

- Points only, snake draft
- Every guide will be simulated in every draft position
- Six different draft lengths (10, 15, 20, 25, 30, and maximum number of rounds)
- A draft order scenario ends as soon as one list is exhausted of its players and the next round cannot be completed
- No positional restrictions
- All players count towards the team total
- Removal of all known retired, non-NHLers, junior returned, and LTIR players at the start of the season

-----
Is there any negative control? Yes

In other seasons I have used lists by Last Year's by Points, Last Year's by Points per Game, Three Year's by Points per Game, and Three Year's by Points.
Last Year by Points is the best of the four lists, but it never wins a whole bunch. It tends to be in last place or near the bottom.
I use these lists as filler if I don't have a multiple of 8 or 9 for a grouping.

-----
Is there any positive control? There was.

I one of my very early videos I made a consensus of the guides I collected that season and it rated highly, but did not win the championship.
However, my goal isn't to evaluate if the consensus wins or not (at least not my goal at the moment). So I have purposely decided to only compare guides/lists/draft kits/cheat sheets to themselves.

-----
I agree guides provide more than projections, but I am evaluating the projections lists. Some sites/people only provide a list and no other commentary. They should be included in any evaluation.

-----
The people who create their projections lists are the ones who indicate what guides help them build their own lists. Most are very up front about it, but some others don't disclose what is in their "secret sauce". If people want to find out more about those guides/lists then they can search the internet and find out what they are all about.

-----

If you feel this is useful for your own knowledge then great.
If you feel this has some value but not fully developed at the moment, then keep on watching. I plan to do more testing for this year's set of lists.
If you feel that this has no value at all, then thank you for reading up to this point.