The JamesâÂÂStein estimator is an estimator of the mean for a multivariate random variable .
It arose sequentially in two main published papers. The earlier version of the estimator was developed in 1956, when Charles Stein reached a relatively shocking conclusion that while the then-usual estimate of the mean, the sample mean, is admissible when , it is inadmissible when . Stein proposed a possible improvement to the estimator that shrinks the sample mean towards a more central mean vector (which can be chosen a priori or commonly as the "average of averages" of the sample means, given all samples share the same size). This observation is commonly referred to as Stein's example or paradox. In 1961, Willard James and Charles Stein simplified the original process.
It can be shown that the JamesâÂÂStein estimator dominates the "ordinary" least squares approach in the sense that the JamesâÂÂStein estimator has a lower mean squared error than the "ordinary" least squares estimator for all . This is possible because the JamesâÂÂStein estimator is biased, so that the GaussâÂÂMarkov theorem does not apply.
Similar to the Hodges' estimator, the James-Stein estimator is superefficient and non-regular at .
Let where the vector is the unknown mean of , which is -variate normally distributed and with known covariance matrix .
We are interested in obtaining an estimate, , of , based on a single observation, , of .
In real-world application, this is a common situation in which a set of parameters is sampled, and the samples are corrupted by independent Gaussian noise. Since this noise has mean of zero, it may be reasonable to use the samples themselves as an estimate of the parameters. This approach is the least squares estimator, which is .
Stein demonstrated that in terms of mean squared error , the least squares estimator, , is sub-optimal to shrinkage based estimators, such as the JamesâÂÂStein estimator, . The paradoxical result, that there is a (possibly) better and never any worse estimate of in mean squared error as compared to the sample mean, became known as Stein's example.
If is known, the JamesâÂÂStein estimator is given by
James and Stein showed that the above estimator dominates for any , meaning that the JamesâÂÂStein estimator has a lower mean squared error (MSE) than the maximum likelihood estimator. By definition, this makes the least squares estimator inadmissible when .
Notice that if then this estimator simply takes the natural estimator and shrinks it towards the origin 0. In fact this is not the only direction of shrinkage that works. Let be an arbitrary fixed vector of dimension . Then there exists an estimator of the JamesâÂÂStein type that shrinks toward , namely
The JamesâÂÂStein estimator dominates the usual estimator for any . A natural question to ask is whether the improvement over the usual estimator is independent of the choice of . The answer is no. The improvement is small if is large. Thus to get a very great improvement some knowledge of the location of is necessary. Of course this is the quantity we are trying to estimate so we don't have this knowledge a priori. But we may have some guess as to what the mean vector is. This can be considered a disadvantage of the estimator: the choice is not objective as it may depend on the beliefs of the researcher. Nonetheless, James and Stein's result is that any finite guess improves the expected MSE over the maximum-likelihood estimator, which is tantamount to using an infinite , surely a poor guess.
Seeing the JamesâÂÂStein estimator as an empirical Bayes method gives some intuition to this result: One assumes that itself is a random variable with prior distribution , where is estimated from the data itself. Estimating only gives an advantage compared to the maximum-likelihood estimator when the dimension is large enough; hence it does not work for . The JamesâÂÂStein estimator is a member of a class of Bayesian estimators that dominate the maximum-likelihood estimator.
A consequence of the above discussion is the following counterintuitive result: When three or more unrelated parameters are measured, their total MSE can be reduced by using a combined estimator such as the JamesâÂÂStein estimator; whereas when each parameter is estimated separately, the least squares (LS) estimator is admissible. A quirky example would be estimating the speed of light, tea consumption in Taiwan, and hog weight in Montana, all together. The JamesâÂÂStein estimator always improves upon the total MSE, i.e., the sum of the expected squared errors of each component. Therefore, the total MSE in measuring light speed, tea consumption, and hog weight would improve by using the JamesâÂÂStein estimator. However, any particular component (such as the speed of light) would improve for some parameter values, and deteriorate for others. Thus, although the JamesâÂÂStein estimator dominates the LS estimator when three or more parameters are estimated, any single component does not dominate the respective component of the LS estimator.
The conclusion from this hypothetical example is that measurements should be combined if one is interested in minimizing their total MSE. For example, in a telecommunication setting, it is reasonable to combine channel tap measurements in a channel estimation scenario, as the goal is to minimize the total channel estimation error.
The JamesâÂÂStein estimator has also found use in fundamental quantum theory, where the estimator has been used to improve the theoretical bounds of the entropic uncertainty principle for more than three measurements.
An intuitive derivation and interpretation is given by the Galtonian perspective. Under this interpretation, we aim to predict the population means using the imperfectly measured sample means. The equation of the OLS estimator in a hypothetical regression of the population means on the sample means gives an estimator of the form of either the JamesâÂÂStein estimator (when we force the OLS intercept to equal 0) or of the Efron-Morris estimator (when we allow the intercept to vary).
Despite the intuition that the JamesâÂÂStein estimator shrinks the unbiased least-squares estimator toward , the estimator actually moves away from for small values of as the multiplier on is then negative. This can be remedied by replacing this multiplier by zero when it is negative. To this end, define the positive-part James-Stein shrinkage operator:
where , and apply this operator component-wise to the (unbiased) least-squares estimator of (with known ) for each :
The resulting estimator of is called the positive-part JamesâÂÂStein estimator and can be written in vector notation as:
This estimator has a smaller risk than the basic JamesâÂÂStein estimator for . It follows that the basic JamesâÂÂStein estimator is itself inadmissible.
It turns out, however, that the positive-part estimator is also inadmissible. This follows from a more general result which requires admissible estimators to be smooth.
Recall the initial setup:
where the variance coefficient is known and we wish to estimate the unknown (mean response) coefficient . In the more general setting of linear regression, the mean response is instead given by
where is a matrix with columns. As in the previous section, we can use the positive-part James-Stein shrinkage operator to obtain a shrinkage estimator of . In particular, any that satisfies the James-Stein KKT conditions:
is a (positive-part) James-Stein estimator of with the useful property that it performs both shrinkage and model selection simultaneously. This is because, depending on the value of the known , there is a (possibly empty) set such that
In other words, some (or all) of the could be estimated as exactly zero, which is equivalent to the selection of a suitable linear regression model.
The JamesâÂÂStein estimator may seem at first sight to be a result of some peculiarity of the problem setting. In fact, the estimator exemplifies a very wide-ranging effect; namely, the fact that the "ordinary" or least squares estimator is often inadmissible for simultaneous estimation of several parameters. This effect has been called Stein's phenomenon, and has been demonstrated for several different problem settings, some of which are briefly outlined below.