During a thread on one of the Community Prospect List posts a couple weeks ago, a few contributors noted a potential flaw in the methodology of our approach to finding a "community" top prospect list. There are a couple of complaints that arise every year when we do this community project, foremost being this: by having a poll attached to the posts, anyone can vote, even people who aren't members of the blog. Further, one could vote and then not bother to defend their selection, or indeed even read the comment threads debating the merits of one prospect over another.
As I said, this is a fairly common thing to hear when running a project like this. Over at John Sickels' site, Minor League Ball, they have a different method of running their community list: instead of a poll, the moderator lists, in a comment, each of the nominated prospects, and votes are garnered by replying to that comment with a "+1", then the plus-one's are added up and a winner decided. I, and many others, wondered if such a system would be better or worse than the one we use now. As far as I know the reasons we use the voting system we use are two: it's easy, and it's the way we've always done it.
So we came up with an idea. I had everyone who was interested in participating email me their top 15 prospects list. Then, using a points system where a #15 vote got 1 point and a #1 vote got 15 points, I tabulated the scores of each prospect and ranked them. The results were quite interesting.
First, the results. The top 15 in the Community Prospect List looks like this:
1. Kyle Crick
2. Gary Brown
3. Clayton Blackburn
4. Chris Stratton
5. Joe Panik
6. Mike Kickham
8. Mac Williamson
9. Adalberto Mejia
10. Gustavo Cabrera
11. Martin Agosta
12. Edwin Escobar
13. Chris Heston
14a. Francisco Peguero
14b. Andrew Susac
Now, the results under this altered system.
Rank, Name, Avg. Points, High Rank, Low Rank
1. Kyle Crick, 14.25, 1, 2
2. Gary Brown, 13.44, 1, 6
3. Chris Stratton, 13.125, 1, 4
4. Clayton Blackburn, 12.81, 1, 5
5. Joe Panik, 10.875, 4, 7
6. Mike Kickham, 9.69, 5, 9
7. Mac Williamson, 7.00, 5, NR
8. Adalberto Mejia, 6.69, 7, 12
9. Heath Hembree, 6.19, 6, NR
10. Gustavo Cabrera, 5.00, 5, NR
11. Edwin Escobar, 4.50, 7, NR
12. Martin Agosta, 4.19, 7, NR
13. Andrew Susac, 3.00, 9, NR
14. Chris Heston, 2.75, 7, NR
15. Francisco Peguero, 1.81, 7, NR
A few observations:
- The sample size on this poll was 16 voters, which obviously isn't very big. I believe all the regular prospect mavens we've come to know and love chimed in, including Fla-Giant, Roger, shankbone, Lyle, Evan, etc. etc.
- We have the exact same 15 names in the top 15 by each method, and indeed the lists are very similar. The only player to have a difference of more than one spot between the two lists is Heath Hembree, who was seventh on the standard CPL and ninth on this new version.
- The points system allows tiers to become visible. Roughly, we have the top four, then a sizable gap, then Panik, then a smaller gap, then down the list in a relatively even manner.
- It's worth noting that #16 on both lists was Adam Duvall, and #17 on both lists was Josh Osich.
- Since many people didn't send me their lists until we were in the 12-15 range on the CPL, it is worth noting that the first list informs the other; that is, that the debates we had choosing the CPL changed people's minds about certain prospects, causing them to be bumped up and down various people's lists. I don't think this is necessarily a bad thing, but it's worth noting.
- Conclusions? I don't really have any, draw your own. I happen to think that the method of voting that we currently use is fine, and it is certainly a hell of a lot less work. But it was an interesting experiment to run. We talk about things like this every year, so it was good to actually run through the process once to take a look at what the results would look like.