A. M. KAGAN

ON AN EMPIRICAL BAYESIAN APPROACH TO THE ESTIMATION PROBLEM

(Presented by Academician V. I. Smirnov on 25 VI 1962)

Let \(X\) be a random variable with a countable set of possible values \(x_1, x_2, \ldots\). Suppose that the distribution of the random variable \(X\) depends in a known way on a parameter \(\alpha \in A\):

\[ p(x_i;\alpha)=P(X=x_i\mid \alpha), \]

\[ P_\alpha=\{p(x_1;\alpha),\ p(x_2;\alpha),\ldots\}. \]

Suppose further, as is done in the Bayesian approach, that the parameter \(\alpha\) is also a random variable, i.e., that on some \(\sigma\)-algebra \(\mathfrak A\) of subsets of the parameter set \(A\) there is specified an a priori distribution \(G(A)\), \(A\in\mathfrak A\). Then the (unconditional) distribution of the random variable \(X\) will be

\[ p_G(x_i)=\int_A p(x_i;\alpha)\,dG(\alpha), \tag{1} \]

\[ P_G=\{p_G(x_1),p_G(x_2),\ldots\}. \tag{2} \]

Below we shall assume \(A\) to be a Borel subset of the line and \(\mathfrak A\) the \(\sigma\)-algebra of Borel subsets of \(A\).

If in an experiment consisting of observation of the random variable \(X\) it has taken the value \(x\), then

\[ \hat{\alpha}(x)= \frac{\int_A \alpha p(x;\alpha)\,dG(\alpha)} {\int_A p(x;\alpha)\,dG(\alpha)} \tag{3} \]

is an estimate of the unknown value of the parameter \(\alpha\) in the experiment, possessing the following property:

\[ E[\hat{\alpha}(x)-\alpha]^2 \le E'[\varphi(x)-\alpha]^2 \]

for all measurable \(\varphi(x)\).

However, the Bayesian estimate \(\hat{\alpha}(x)=E(\alpha\mid x)\) of the parameter can be used only when the a priori distribution \(G\) is known. But in a number of applied problems (some of them are indicated in \((^4)\)) \(G\) is unknown, and the situation is such that by the time \(\alpha\) is to be estimated from the results of the experiment, the statistician has at his disposal a sufficiently long series of independent observations of the random variable \(X\), obtained for unknown values of the parameter \(\alpha\), which has an (unknown) distribution law \(G\).

Problems of this kind were first pointed out in \((^1)\); therefore the experimental scheme described in the preceding paragraph will be called Robbins’ scheme. In \((^1)\) it is also shown that for some distributions \(P_\alpha\) (for example, the Poisson distribution,

\[ p(x;\alpha)=\frac{e^{-\alpha}\alpha^x}{x!},\quad x=0,1,2,\ldots) \]

independent observations of \(X\) obtained according to Robbins’ scheme can be used for consistent estimation of \(E(\alpha\mid x)\). In \((^2)\) a consistent estimate of \(E(\alpha\mid x)\) is constructed from independent observations according to Robbins’ scheme of a random variable \(X\) having a normal distribution with known variance and unknown mean \(\alpha\) as parameter.

In the present note we report general results concerning consistent estimation of \(E(\alpha\mid x)\) from independent observations on a discrete random variable \(X\) according to Robbins’ scheme.

Introduce in the space \(\mathcal P\) of sequences \(P_G\) the metric \(\rho\) as follows:

\[ \rho(P_{G_1}, P_{G_2})=\sup_{x_i}\left|p_{G_1}(x_i)-p_{G_2}(x_i)\right|. \]

Theorem 1. In order that there exist a consistent estimate of \(E(\alpha\mid x)\) from independent observations on \(X\) according to Robbins’ scheme, it is necessary and sufficient that

\[ \int_A \alpha p(x;\alpha)\,dG(\alpha)=\lim_{n\to\infty} F_n(P_G;x), \tag{4} \]

where \(F_n(P_G;x)\), \(n=1,2,\ldots\), are continuous functionals on \(\mathcal P\).

Theorem 1 is very close to the main theorem of paper \((^3)\).

We shall say that \(P_\alpha\) satisfies condition E if from

\[ \int_A p(x_i;\alpha)\,dG_1(\alpha) = \int_A p(x_i;\alpha)\,dG_2(\alpha) \quad \text{for } i=1,2,\ldots \]

it follows that \(G_1=G_2\).

Theorem 2. If: 1) \(P_\alpha\) satisfies condition E; 2) for all \(i=1,2,\ldots\), \(p(x_i;\alpha)\) is continuous in \(\alpha\), then consistent estimation of \(E(\alpha\mid x)\) from independent observations on \(X\) according to Robbins’ scheme is possible.

We now give a simple example of a family \(P_\alpha\) for which consistent estimation of \(E(\alpha\mid x)\) from observations on \(X\) according to Robbins’ scheme is impossible\(^*\).

\(A=[0,1]\), \(n\) is any natural number;

\[ p(x;\alpha)=C_n^x \alpha^x(1-\alpha)^{n-x} \tag{5} \]

for \(x=0,1,\ldots,n\).

One can choose distributions \(G_1\) and \(G_2\) on \([0,1]\) so that for all \(x=0,1,\ldots,n\)

\[ \int_0^1 \alpha^x(1-\alpha)^{n-x}\,dG_1(\alpha) = \int_0^1 \alpha^x(1-\alpha)^{n-x}\,dG_2(\alpha); \tag{6} \]

\[ \int_0^1 \alpha^{x+1}(1-\alpha)^{n-x}\,dG_1(\alpha) \ne \int_0^1 \alpha^{x+1}(1-\alpha)^{n-x}\,dG_2(\alpha). \tag{7} \]

Conditions (6) and (7) are easily satisfied by choosing distributions \(G_1\) and \(G_2\) such that all their moments up to moments of order \(n\) coincide, while the moments of order \(n+1\) are different.

It is obvious that the family of distributions (5) does not satisfy the conditions of Theorem 1, and consistent estimation of \(E(\alpha\mid x)\) in Robbins’ scheme is impossible.

The author expresses gratitude to Yu. V. Linnik for his attention to this work.

Received
19 VI 1962

REFERENCES

\({}^1\) H. Robbins, Proc. III Berkeley Symposium on Math. Statistics and Probability, 1, 1956.
\({}^2\) K. Miyasawa, Bull. de l’Inst. Intern. de Statistique, 38 (1961).
\({}^3\) L. Le Caen, L. Schwartz, Ann. Math. Statistics, 31, 1 (1960).
\({}^4\) J. Neyman, Two Breakthroughs in the Theory Statistical Decision Making, Report at the Statistical Laboratory in Berkeley, 1961.

\(^*\) We draw attention to the ambiguity of the assertion contained in (1) that for the family of distributions defined by (5), consistent estimation of \(E(\alpha\mid x)\) in Robbins’ scheme is possible.