SOME LIMITING INFORMATION RELATIONS IN PREDICTION AND REGRESSION PROBLEMS

R. N. LYUBLINSKY

(Presented by the scientific seminar of the Department of Automation and Telemechanics)

In the operational control of complex production complexes, problems arise of predicting the state of individual objects at future moments in time, of regression estimation of current and future values of object parameters from observations of other parameters and objects, and of individual prediction of gradual equipment failures. In controlling development processes, prediction is one of the necessary elements of decision-making [1].

At present there is a large number of works on the synthesis of optimal prediction and regression operators for various situations, namely: for stationary and nonstationary useful signal \(x(t)\) and noise \(f(t)\); for processes \(x(t)\) and \(f(t)\) with different distribution laws; for discrete and continuous observations of the processes \(x(t)\); for different types of distortions introduced in measurements, etc. [2]. Methods of prediction and regression, according to the a priori information used for prediction and according to the degree of connection of this information with the estimated state vectors \(z\), can be divided into two groups: a) methods based on objective models of the development of the phenomenon being predicted; b) methods for obtaining estimates by means of certain systems of questioning specialists, the so-called expert estimates.

The most widespread are prediction methods based on a probabilistic mathematical model of the development of the phenomenon being predicted. These include methods based on pattern recognition, statistical modeling, the solution of diffusion equations, methods establishing filtering of the type of filters of N. Wiener and R. Kalman, and methods for obtaining regression estimates. An important question in obtaining regression and predictive estimates is the analysis of their reliability and of the conditions under which the reliability will be maximal.

In the present work, limiting information estimates of the reliability of discrete prediction and regression operators are obtained in the most general form, i.e., without restrictions on the nature of the initial process and on the nature of the distortions introduced by the measuring instrument.

The performance of predictive and regression estimates on the basis of a probabilistic dynamic model is explained by the following structural scheme (Fig. 1).

$\bar X$ is a segment of a realization of the processes of development of the phenomenon being predicted, used to obtain estimates; $y$ is the control data of the processes $x$; $f$ are disturbing processes that cause measurement errors; $z$ is the state vector being estimated; $\varepsilon$ is the vector of the error in estimating the state; $W$ is the operator for carrying out the estimate.

Fig. 1

The state vector $z$ being estimated is, respectively:

a) in the predictive estimate of the state of an object at a future time from its prehistory, specified by observations of the vector process $x(t)$ on the time interval $(t_{k-m}, t_k)$,

\[ z = x(t_k + T), \]

where $T$ is the prediction time interval;

b) in the regression estimate of the vector process $N(t)$ from observation data $\bar y$ on the values of another vector process $x(t)$,

\[ z = N(t) = \varphi(\bar x); \]

c) in individual prediction of gradual failures from observation data $\bar y$ on the processes $x(t)$ that lead to failures,

\[ z = t_{\mathrm{fail}}, \]

where $t_{\mathrm{fail}}$ is the time of the first departure of the vector process $x(t)$ beyond the limits of admissible values. The error vector $\varepsilon(t)$ is formed by comparing the estimate of the vector $\hat z$, obtained by the action of the operator $W$ of the real device on the control data $\bar y$, with the vector $z$, which represents the ideal estimate, i.e.

\[ \varepsilon = z - \hat z . \tag{1} \]

The magnitude of the mean reduction of entropy when performing the estimate, i.e., the average amount of information (AAI) obtained thereby, is determined by the following expression:

\[ I_0 = H(z) - H(\varepsilon), \tag{2} \]

where $H(z)$ is the a priori entropy of the vector $z$; $H(\varepsilon)$ is the entropy of the error vector $\varepsilon$.

Existing methods for synthesizing the optimal operator $W$ are, in the overwhelming majority of cases, based on minimizing the mean square (variance) of the error $\sigma_\varepsilon^2$ or functions of various moments of the error distribution law [2]. There are, however, a number of works in which

as the minimized optimality criterion for filtering, the entropy of the error vector \(H(\varepsilon)\) is considered [5, 6].

Let us consider the entropy \(H(\varepsilon)\) as an optimality criterion. From expression (2) it follows that minimization of the quantity \(H(\varepsilon)\) is equivalent to maximization of the quantity \(I_0\). We shall study the limiting properties of the quantity \(I_0\). In doing so we shall use the limiting relations for communication channels and for estimating the parameters of distribution laws [4, 5].

Theorem 1. The average amount of information \(I_0\) obtained in performing a predictive or regression estimate of the state of an object is bounded above by the value of the average amount of information \(I(\bar y \to z)\) about the estimate vector \(z\) contained in the monitoring data \(\bar y\) of the process \(\bar x\) used for the estimate, i.e. \(I_0 \leq I(\bar y \to z)\).

Proof. The quantity \(I_0\) is determined by expression (2). Let us transform this expression so as to obtain the quantity \(I(\bar y \to z)\) on the right-hand side:

\[ I_0 = H(z) - H(\varepsilon) = H(z) - H(z/\bar y) + H(z/\bar y) - H(\varepsilon), \tag{3} \]

where \(H(z/\bar y)\) is the conditional entropy of the vector \(z\) in the presence of the monitoring data \(\bar y\).

Since \(H(z) - H(z/\bar y) = I(\bar y \to z)\), equation (3) is transformed into the form

\[ I_0 = I(\bar y \to z) + H(z/\bar y) - H(\varepsilon). \tag{4} \]

The quantity \(\hat z\), in accordance with the operator \(W\), is uniquely determined from the monitoring data \(\bar y\). Consequently, for the probability distribution densities of the vector \(z\) and the process \(\bar y\), the following must hold:

\[ f(z/\bar y) = f[\varepsilon + \hat z/\bar y] = f(\varepsilon/y). \]

It follows that

\[ H(z/\bar y) = H(\varepsilon/\bar y). \tag{5} \]

Consequently,

\[ H(z/\bar y) - H(\varepsilon) = H(\varepsilon/\bar y) - H(\varepsilon) = - I(\bar y \to \varepsilon), \tag{6} \]

where \(H(\varepsilon/\bar y)\) is the conditional entropy of the vector \(\varepsilon\); \(I(\bar y \to \varepsilon)\) is the average amount of information about the vector \(\varepsilon\) contained in the monitoring data \(\bar y\).

From expressions (4) and (6) it follows that

\[ I_0 = I(\bar y \to z) - I(\bar y \to \varepsilon). \tag{7} \]

By virtue of the nonnegativity property of the amount of information,

\[ I(\bar y \to \varepsilon) \geq 0, \]

and therefore

\[ I_0 \leq I(\bar y \to z). \]

The average amount of information \(I_0\) coincides with its upper limit \(I(\bar y \to z)\) if \(I(\bar y \to \varepsilon)=0\), i.e., under statistical independence of the error vector \(\varepsilon\) and the process \(\bar y\), which testifies to the complete extraction of the information of interest to us from the data \(\bar y\).

Since it is always true that

\[ I(\bar y \to z) < H(z), \]

there exists a residual entropy

\[ H_0 = H(z) - I(\bar y \to z), \]

which is the lower limit of the entropy reduction when perform-

of the estimate. As a consequence, the error vector \(\varepsilon\) can be represented as the sum of two components: \(\varepsilon_{\mathrm{n}}\), due to the randomness of the vector \(z\), and \(\varepsilon_y\), due to errors of the estimation method. Improvement of the estimation operator leads to a reduction of the component \(\varepsilon_y\). The ideal estimate is an operator for which \(\varepsilon_y=0\).

Corollary. The average amount of information \(I_0\) obtained by carrying out predictive or regression estimates, under ideal measurement conditions, does not exceed the average amount of information \(I(x \to z)\) of the estimate vector \(z\) contained in the processes \(x\) used for estimation, i.e.

\[ I_0 \leq I(x \to z). \]

Proof. When measurements are made, errors caused by the influence of the environment and by the properties of the measuring instruments are imposed on the process \(x(t)\). Therefore, an ideal measuring device is one in which the measured process is transmitted without distortion to the output of the device, i.e. \(y=x\), and then the structural diagram (Fig. 1) is transformed into the following form (Fig. 2).

Fig. 2

In accordance with Theorem 1, in this case \(I_0 \leq I(x \to z)\). Consequently, the maximum decrease in entropy under an ideal measuring device and an ideal operator for carrying out estimates does not exceed the value of the average amount of information \(I(x \to z)\).

Theorem 2. Optimization of the operator \(W\) according to the criterion of the maximum average amount of information obtained in estimation is equivalent to minimization of the quantity of mutual information \(I(y \to \varepsilon)\) between the control data \(y\) used for estimation and the error vector \(\varepsilon\).

Proof. From expression (7),

\[ I_0 = I(y \to z) - I(y \to \varepsilon). \]

The quantity \(I(y \to z)\) is determined by the interrelation of the process \(y\) and the vector \(z\) and does not depend on the properties of the operator \(W\). Therefore, maximizing the quantity \(I_0\) by an appropriate choice of the operator \(W\) is equivalent to minimizing the quantity \(I(y \to \varepsilon)\).

The theorem and the corollary determine the limiting values of the average amount of information obtained in carrying out predictive or regression estimates. It is evident that these limiting values of the quantity \(I_0\) will be obtained when synthesizing the operator \(W\) optimal according to the criterion of minimum entropy \(H(\varepsilon)\). For a normal probability distribution of the vector \(\varepsilon\), the value \(I_0\) obtained by optimizing the operator \(W\) according to the criterion of minimum variance of each coordinate of the vector \(\varepsilon\) coincides with the limiting value \(I(y \to z)\), i.e. such optimization is equivalent to optimization according to the criterion of minimum entropy. The quantity \(H(z)\) is determined by the following expression

\[ H(z)=-\int_{-\infty}^{\infty}\cdots\int_{-\infty}^{\infty} f(z)\log f(z)\,dz, \tag{8} \]

where \(f(z)\) is the multivariate probability density of the coordinates \(z\).

For continuous-time processes \(\bar{x}\) and \(\bar{y}\), the quantities \(I(\bar{y}\to z)\) and \(I(\bar{x}\to z)\) are defined as the supremum of the multivariate integral dependence. Therefore, in order to simplify the computational formulas, one may quantize the processes \(\bar{x}\) and \(\bar{y}\) in time and regard them as sequences of random variables. Even when the conditions of V. A. Kotelnikov’s theorem are satisfied, such a consideration is an approximation because of the finiteness of the time interval under consideration \((t_{k-m}\div t_k)\). However, such an approximation is admissible. Moreover, in practice the monitoring of the process \(x(t)\) is most often carried out discretely. If the process \(\bar{y}\) is considered as a discrete sequence of random vectors, then the expression \(I(\bar{y}\to z)\) has the form

\[ I(\bar{y}\to z)=\int_{-\infty}^{\infty}\cdots\int_{-\infty}^{\infty} f(\bar{y},z)\log \frac{f(\bar{y})\cdot f(z)}{f(\bar{y},z)}\,d\bar{y}\cdot dz, \tag{9} \]

where \(f(\bar{y},z)\) is the multivariate joint probability density of the random sequence \(\bar{y}\), considered at the time instants \(t_{k-m}, t_{k-m+1}, \ldots, t_k\), and of the vector \(z\); \(f(\bar{y})\) is the multivariate distribution density of the random sequence \(\bar{y}\).

Expressions (8) and (9) can be substantially simplified by replacing them with approximate expressions based on the correlation matrices of the sequence \(\bar{y}\) and the vector \(z\)

\[ H(z)\approx \frac{1}{2}\log_2(2\pi l)^n \det A, \tag{10} \]

where \(A=\|R_{z_i z_j}\|\), \(i,j=\overline{1,n}\); \(R_{z_i z_j}\) is the cross-correlation moment of the coordinates \(z_i\) and \(z_j\) of the vector \(z\).

\[ I(\bar{y}\to z)\approx \frac{1}{2}\log_2\frac{\det A\cdot \det B}{\det C}, \tag{11} \]

where \(B\) and \(C\) are the matrices of the correlation relationship of the coordinates of the random sequence \(y\) and the vector \(z\) [7]. Expression (10) is exact for the normal probability distribution law of the vector \(z\).

Expression (11) is exact for normal distributions of the vector \(z\) and the random sequence of vectors \(y\). For other distributions, expression (11) represents a somewhat overestimated quantity.

LITERATURE

1. S. V. Emel’yanov, E. B. Dudin, A. A. Malevich. Problems of controlling processes of development of production, science, and technology and modern methods for their investigation. In: “Technical Cybernetics.” 1968. Ed. by B. N. Petrov, Moscow, 1970.

2. N. I. Andreev. Correlation Theory of Statistical Optimal Systems. Moscow, “Nauka,” 1966.

3. K. E. Shannon. Mathematical theory of communication. In: “Works on Information Theory and Cybernetics.” Moscow, IL, 1963.

4. N. L. Weideman and E. B. Stear. Entropy Analysis of Estimating Systems.—IEEE Transaction on Information Theory vol. 11—16, No. 3, 1970.

5. G. G. Voronina. Criteria of optimization in measurement engineering and control engineering from the standpoint of information theory. In: “Problems of Electrometry.” Novosibirsk, “Nauka,” 1967.

6. B. N. Petrov, V. V. Petrov et al. The beginning of the information theory of control. In: “Technical Cybernetics. 1968.” Ed. by B. N. Petrov, Moscow, 1970.

7. R. N. Lyublinskii. Information analysis of the state of a dynamic system. “Proceedings of LETI named after V. I. Ulyanov (Lenin),” issue 90, 1970.

8. Yu. V. Prokhorov, Yu. A. Rozanov. Probability Theory. A Handbook. Moscow, “Nauka,” 1967.