One of my favorite topics on the McCovey Chronicles is Minorlines and the debate on the potential of Giants prospects, but since I live on the east coast and nowhere near Richmond or Augusta I’m limited in my ability to assess prospects via the oldfashioned eye test. Aside from a few college AtBats and the scouting videos on the internet, I suspect I’m like most of us, I get my information from the typical sources, Baseball America, John Sickels et. al. I’ve often thought it would be cool to have a sabermetric system of grading prospects, so I finally got off my butt and did what any industrious McCoven would do an made one.
So, here’s The Blind Baseball Scout, which we’ll refer to as PSABR (if you can think of something better, please let me know). Today we’ll look at some historical comparisons of PSABR versus Baseball America’s top hitters (there’s no PSABR for Pitchers yet) by league (AZ, NW, SALLY, CAL and EL) from 2003 to 2008.
I should note that PSABR is meant to be a quick and dirty analysis tool, and has some obvious drawbacks. It’s compiled using players with the top 100 plate appearances in the league, so some “hot” prospects who are promoted midseason don’t qualify for PSABR due to its sample size constraints. An example of this in 2011 would be Bryce Harper, who made two lists for Baseball America, but did not qualify for PSABR due to too few plate appearances in both the Sally and Eastern League.
Here are some significant players selected by Baseball America between 20032008 who did not qualify for PSABR due to too few plate appearances.


While this sample size demand appears to be a considerable limitation, PSABR seems to be adept at finding players that Baseball America overlooks as well. Here are some players that appear on PSABR’s rankings, but did not make Baseball America’s top lists (by league).
Players found by PSabr, Not BBA 
Year 
League 
2005 
Sally 

2003 
Cal 

2005 
EL 

2004 
Sally 

2007 
Sally 

2004 
Sally 

2007 
NWL 

2004 
Sally 

2005 
EL 

2006 
Sally 

Michael Brantley* 
2005 
AZL 
2003 
EL 

2004 
EL 

Nick Markakis 
2004 
Sally 
2004 
EL 
There are also players that PSABR just did not rate high enough to make its list. The first thing you’ll notice about these eleven players is that 5 of them are (or in the case of Pablo, were) catchers, others like Span, Bourne and Bourjos owe a good to significant proportion of their value to their defensive skills, which represent two areas of potential improvement to the PSABR system, Positional and defensive valuations. As it stands now, PSABR has no way of accounting for a prospect potential value based on position or defensive skill.
2003 
AZL 

2003 
Sally 

2005 
NWL 

Chris Ianetta 
2005 
Cal 
2005 
EL 

Michael Bourne 
2005 
EL 
2005 
Cal 

2005 
NWL 

Pablo Sandoval 
2004 
AZL 
2008 
Cal 

2004 
EL 
Over the next couple of days we’ll look at comparisons of Giants prospects over the last few years, and then take a look at what PSABR thinks of Giants prospects of 2011. But first here’s the statistical paper that analyzes PSABR, complete with hypothesis testing and explanations of the grading system – so be warned –Geeks only!
The Blind Baseball Scout
Abstract
The Blind Baseball Scout (PSABR) is a Sabermetric model that measures minor league production with the goal of predicting a players future value as, and likelihood to become a Major League Ballplayer. The PSABR assigns points to players across 9 different statistical areas, the sum of which produces a total score for the entire season. The top players represent the players with the highest scores. In order to test this model the results have been compared against the top hitting prospects from Baseball America (BBA), the nations leading scouting publication, using the proportion of players who make the Major Leagues and the population of “top players” mean WAR value as measurements of value from both systems. PSABR was run from 2003 to 2008 in five select leagues, the Arizona Rookie league, The Northwest Rookie League, The Sally League, The California League and the Eastern League and compared to BBA’s results. The results of both systems were analyzed statistically on the basis of proportion of players who made the major leagues. The populations of both methods were then tested based on their mean “WAR” value (based on WAR {Wins over replacement}, a Sabermetric valuation of individual players). Both comparisons were tested by the hypothesis that PSABR would be equal to BBA in finding future major leaguers and forecasting the value of those players at a 98% confidence level, with the alternative hypothesis being that they will not be equal.
Introduction
The goal of this project is to determine a viable process in applying the use of sabermetric data in projecting the future performance and value of minor league prospects.
Sabermetrics is generally considered an effective and objective method in the valuation of Major League Baseball player performance, however because of a number of factors it is not nearly as reliable in predicting the future performance of Minor League players. This process will attempt to address the factors that make the application of sabermetrics to minor league performance ineffective by giving specific skills, flaws and factors greater weight in order to bring equilibrium to the players performance.
In order to measure the effectiveness of this system, I will test the hypothesis that this system is equal Baseball America, the nations leading scouting publication, by comparing their top hitting prospects from 2003 to 2008 in five select leagues, the Arizona Rookie league, The Northwest Rookie League, The Sally League, The California League and the Eastern League to this system. I will compare proportions of ranked players who go on to see Major League playing time in both systems, as a test of proportion. I will also compare the value of those players who made it to the major leagues based on their Offensive WAR (Wins Above Replacement) value as a test of means. In this later test, the mean (WAR value) of both systems will be taken for both systems.
The aforementioned variables that make Minor League predictions so difficult are age, where “very young players, as a whole, return 25 percent more value than expected by their draft slots” (Rany Jazayerli), Strikeout rates , where “it appears as though the success rates for prospect development drop sharply when strikeout rates hit about 22%.” (Minorleagueball.com). Then there are the more traditionally acknowledged variables like Walk Rates, Batting Average, ExtraBase Hit percentage, Home Run Percentage, On Base Percentage. There’s also my own metric Stolen Base Efficiency, which attempts to project the value of a hitters speed and running ability. The best performers in each category are assigned a score, which decreases towards zero incrementally as we reach the median, once the median is reached scores progress negatively incrementally towards the worst performers.
The data will be obtained from Baseballreference minor league data sorted by Plate Appearances. The top 100 plates appearances will be each sample. This will be used to ensure the largest possible sample sizes. This has the advantage of eliminating top performers who do it over relatively short periods of time, the disadvantage is that sometimes really good prospects spend a short time in a given league because of good performance, are promoted to another league and (consequently) will be undefined under this system.
Notes on the data compilation
Page 1 is the source data compiled from Baseballreference. Com
Page 2 is the age of the players. As mentioned above age is one of the two primary factors in this approach. Previous research from Bill James and Rany Jazayerli are central to the theory, which states, “The younger the player, the greater the slope of the curve—meaning the greater the rate at which he improves” (Javeri, Rani. taken from the web) It is for this reason that age is weighted so heavily in this model.
AGE 
Score 
17 
40 
18 
35 
19 
30 
20 
20 
21 
3 
22 
15 
23 
20 
24 
30 
25 
35 
As you see here players that who are younger than the mean are given points strictly on the basis of the basis of their age. The farther below the mean the greater the positive score, and conversely the greater the age above the mean the greater the negative score.
Players who are at the mean are given a small negative score. This is primarily because they are often (in this league) college players who have experience that is potentially at or above the level of the league.
This weighting system based on age and the weighting system based on K% are the only pieces that will change from league to league. At the lower levels (NW league and AZ league) older players are given a greater disadvantage (negative score), this is based on the theory that their advanced age and experience is an even greater advantage when the competition is so young and inexperienced. Players with higher strike out rates at lower levels are also given a greater disadvantage (negative score) at the lower levels, based on the assumption that the inability to make contact will be a further hindrance at higher levels when the competition is stiffer.
Page 3 is a simple effort to manage sample size. Sample size plays a very important role in this experiment, as it is the basis of the population selected (the top 100 players based on # of Plate Appearances) and it potentially a major weakness in the hypothesis. It is common for organizations to promote their players who are performing well to a higher league, which often mean that those players (good performers) won’t eligible for this method. We will see many players in Baseball America’s (BBA) top rankings who do not qualify for this method.
Mean PA 
468.33 
Plate App. 
Score 
x525 
7 
524475 
3 
474425 
0 
424375 
5 
374x 
10 
A small positive is given for players who have the most PA’s and a small deduction is given to those who have the fewest. This theorizes that the more PA’s one has the better one’s performance will represent ones talent. The mean should fall somewhere in the middle and will be given no bonus or deduction.
Pages 4, 5 and 8 are measurements of one very important measure of value, the ability to not make an out. As Bill James said “the difference between winning teams….is the difference between ‘outs’ and ‘runners on base’ “(James, Bill, The New Bill James Historical Baseball Abstract, 2001, The Free Press. Page 642), these three statistics attempt to measure this skill. First is Batting Average (BA).
<![if !supportEmptyParas]> <![endif]>
Mean BA * 
= 
Score 
1.25 
0.3297125 
17 
1.2 
0.316524 
13 
1.175 
0.30992975 
10 
1.15 
0.3033355 
7 
1.1 
0.290147 
3 
1 
0.26377 
0 
0.95 
0.2505815 
5 
0.9 
0.237393 
10 
0.85 
0.2242045 
15 
0.8 
0.211016 
20 
0.75 
0.1978275 
25 
Averages that are at the mean are credited with 0 and are progressively (i.e. Mu + 10% =3) awarded point as players are above the mean.
The next measurement is on base percentage (OBP), where we see a similar grading system where the highest points are gained above Mu + 20%.

Mu ( x) + Mu 

X 
= 
Score 
0.2 
0.3987 
15 
0.17 
0.3887325 
13 
0.13 
0.3754425 
10 
0.1 
0.365475 
7 
0.07 
0.3555075 
5 
0.03 
0.3422175 
3 
Mu 
0.332 
0 
0.03 
0.3222825 
5 
0.07 
0.3089925 
10 
0.1 
0.299025 
15 
0.13 
0.2890575 
20 
0.17 
0.2757675 
25 
And lastly, there is BB%, which measures the percentage of walks a player takes per plate appearance. “1” represent Mu * 1.
Mu (x) 
= 
Score 
1.65 
0.13645191 
10 
1.4 
0.11577738 
7 
1.25 
0.10337266 
5 
1.1 
0.09096794 
3 
1 
0.08269813 
0 
0.9 
0.07442832 
3 
0.75 
0.0620236 
5 
0.6 
0.04961888 
7 
0.45 
0.03721416 
10 
0.25 
0.02067453 
15 
Next on page 6 and 7 we have two measurements of power hitting ability. Extrabase hit percentage (XBH%) and Home Run percentage (HR%).
XBH%

Mu (x) + Mu 

X 
= 
Score 
0.5 
0.124798452 
15 
0.45 
0.120638504 
13 
0.3 
0.108158658 
10 
0.15 
0.095678813 
7 
0.05 
0.087358916 
3 
Mu 
0.0832 
0 
0.05 
0.07903902 
3 
0.15 
0.070719123 
7 
0.25 
0.062399226 
10 
0.35 
0.054079329 
15 
0.45 
0.045759432 
20 
0.5 
0.041599484 
25 
HR%
X 
Mu (x) = 
Score 
3.5 
0.06884643 
15 
2.75 
0.05409362 
13 
2 
0.03934082 
10 
1.5 
0.02950561 
7 
1.25 
0.02458801 
3 
1 
0.01967041 
0 
0.9 
0.01770337 
3 
0.75 
0.01475281 
7 
0.6 
0.01180224 
10 
0.5 
0.0098352 
15 
0 
0 
20 
On page nine we have Strikeout percentage (K%). Thanks to the work of RedSoxFaithful we have a sense that as Krates in prospects increase, their chance for major leagues success goes down. ( Minorleagueball.com ) This is how we’ll weigh Strike Outs in our model in the higher leagues (Sally, Cal & Eastern).
X 
Mu (x) = 
Score 
0.45 
0.10074501 
15 
0.55 
0.12313279 
13 
0.65 
0.14552057 
10 
0.75 
0.16790835 
5 
0.95 
0.21268391 
0 
1 
0.2238778 
3 
1.1 
0.24626558 
10 
1.2 
0.26865336 
20 
1.35 
0.30223503 
30 
1.5 
0.3358167 
35 
On Page ten we have an attempt to measure speed as a tool for success. This is not really attempt to measure how well one can steal bases, it works on the theory that a player who has potentially valuable speed will be asked to steal often and will have some measured success (Stolen Bases), so this is not the percentage of successful stolen bases, this stolen bases divided by Plate Appearance. The negative grading is significantly less in this instance because, while speed can be a valuable tool, its absence is not necessarily equally a detriment.
X 
Mu (x) = 
Score 
4 
0.11559657 
25 
3.5 
0.101147 
20 
2.75 
0.07947264 
10 
1.75 
0.0505735 
7 
1.35 
0.03901384 
3 
1 
0.02889914 
0 
0.9 
0.02600923 
3 
0.75 
0.02167436 
5 
0.5 
0.01444957 
7 
0.25 
0.00650231 
10 
You’ll notice that in virtually all of these points systems the distribution is weighted heavier on the negative side, or one gets penalized more performing below or at the mean than one gets rewarded for performing above the mean. This is meant to help create a negative linear (or logistical?) model which results in markedly fewer players having a positive rating than players with a negative rating, based on the simple fact that more players will fail to make to the next level than players who move on and eventually become major leaguers. This chart is exemplary of this principle.
(For some reason I am unable to insert my graph here sorry!)
Sample Size
As mentioned before, sample size weighs heavily in this system. Without the benefit eyes to see (a players movement, physicality and swing) and ears to hear (a coaches or scouts observations or fears) PSABR is at significant disadvantage. Sample size represents a factor that we can attempt to account for. As an example, here are the top hitters from the Eastern League 2006 –

BBA Top Hitters 
WAR 
PSABR 
WAR 

1 
4.7 

Carlos Gomez 
1.3 

2 
Jacoby Ellsbury 
DNQ 
12 
1.2 

3 
Carlos Gomez 

1.3 
Kevin Kouzmanoff 
5.1 

4 
DNQ 
1 
Adam Lind* 

4.7 

5 
Kevin Kouzmanoff 

5.1 
0 

6 
1 
Nate Schierholtz 
1.7 

7 
DNQ 
2.6 
0.5 

8 
Kory Casto 

1.4 

9 
Luis Antonio Jimenez* 
0 

10 
Chad Spann 
0 
In this example, only 7 hitters this year made BBA rankings for the top 20 players of the league and 3 of those players (Elsbury, Crowe and Casila) did not qualify for the PSABR system due to an insufficient amount of Plate Appearances. There are two ways we can deal with this, one, remove the players who would not qualify for PSABR and compare only those top players who qualify (4 players), or two, expand the PSABR sample to account for this sample size factor. The later was chosen because it is potentially a better test of PSABR’s value, where potentially PSabr can find a greater proportion of MLB players (n=10) or lesser proportion, as in this case, where BBA had all (7) of their selections make it to the major leagues and PSABR had only 70%. It also expands the population in determining mean value (WAR), where in this case BBA selections have a mean value of 3.36 (WAR) and PSABR a 1.07 (WAR) thus far in their respective careers.
Conclusion
After testing the data it’s evident that PSABR has some real value in finding and projecting the value of minor league players. Two sample Hypothesis tests for proportion and mean were run on each league at a 95% confidence intervals, then run on all the data combined at a 98% confidence interval, where the final P values were .0004 for proportion (rejected proportion) and .0217 (failed to reject mean), suggesting that while BBA is better at finding players who will become MLB players, PSABR finds enough players who have value as future MLB players to suggest that the system has potential.
The null hypothesis of proportional equality was rejected in 2 out of 5 leagues, the California League and the Northwest league as well as in the overall conclusion. Clearly, BBA has an advantage in selecting players who are most likely to become Major league ball players, finding 226 future major leaguers out of a sample of 351, while PSABR found only 213 out of 412.
In the testing of mean, the null hypothesis of equality was rejected once in the Eastern league, but failed to reject overall. This suggests that PSABR has some value in identifying players who may have been overlooked by BBA and their system. The Mean WAR for all players selected by BBA was 2.75, while it was 1.8 for PSABR, suggesting that there’s also room for improvement in the system.
It is also clear that there are areas that PSABR could be improved, most notably a positional ranking, as the players that were most likely overlooked by PSABR had positional values (i.e. catcher or middle infielder) that should have raised their overall value when compared to their peers. The most illustrative example of this would be Brian McCann (Sally league 2003), who ranked 11th in PSABR’s system and 8th in BBA’s, but has gone on to be a 21.7 WAR player thus far in his career.
Since PSABR is limited to analyzing a players potential value as a hitter, the expansion to a more rounded approach is recommendable. One way to attempt to account for a players future defensive value is to include positional rankings, by providing a scale of value for a players defensive position, such as 20 points for a catcher, 10 points for a Shortstop and Center fielder, 5 points for a second baseman, 0 for a Third baseman, 5 Left and Right Fielders and –10 for First baseman.
References
1James, Bill, The New Bill James Historical Baseball Abstract, 2001, The Free Press. Page 642
2(http://www.minorleagueball.com/2011/4/22/2123847/thesignificanceofminorleaguekrates. By Redsoxfaithful
3 http://www.baseballprospectus.com/article.php?articleid=15295#commentMessage
by Rany Jazeyleri.
4 Baseball America, The Baseball America Prospect Handbook 2004, 2004, Simon & Schuster
5 Baseball America, The Baseball America Prospect Handbook 2005, 2005, Simon & Schuster
6 Baseball America, The Baseball America Prospect Handbook 2006, 2006, Simon & Schuster
7 Baseball America, The Baseball America Prospect Handbook 2007, 2007, Simon & Schuster
8 Baseball America, The Baseball America Prospect Handbook 2008, 2008, Simon & Schuster
9 Baseball America, The Baseball America Prospect Handbook 2009, 2009, Simon & Schuster
Hypothesis test of 2 means

Mu 1 
Mu 2 


PSABR WAR 
BBA WAR 

Mu 
1.79782082 
2.75156695 

StDev 
5.0269922 
6.11363269 

Claim µ1 = µ2 

Test Statistic, t: 2.3287 

Critical t: ±2.33186 

PValue: 0.0202 

Degrees of freedom: 677.7948 

98% Confidence interval: 

1.90931 < µ1µ2 < 0.0013097 

Fail to Reject the Null Hypothesis 
< 

Sample does not provide enough evidence to reject the claim 
Connect with McCovey Chronicles