METHODS OF AUTOMATIC FAULT DETECTION AND CONTROL OF COMPLEX SYSTEMS

(review)

V. M. RAZIN, M. N. STROGONOV

(Presented by the scientific seminar of the Faculty of Automation and Computer Engineering)

Introduction

One of the ways to increase the reliability and efficiency of electronic equipment is the use of systems for automatic monitoring and fault detection of the given equipment, combined with the possibility of eliminating the fault.

Complex systems cannot be monitored manually, since much time is required to locate a fault. Automation of monitoring substantially reduces the time spent on inspection and preventive maintenance of equipment; as a result, the useful operating time of the system increases; in addition, the staff of maintenance personnel is reduced and the requirements for their level of qualification are lowered. Taken together, all this leads to a reduction in operating costs. The savings provided by automatic-control systems can be seen from the following examples cited in [10].

The use of the SCATE automatic control system in testing one radio-electronic complex reduced the time of the rocket’s prelaunch check from 12 hours to 5 minutes.

The application of the BOFTE control system made it possible to save 75% of the total time of the prelaunch check of the Terrier rocket, while testing costs were reduced by 97.5%.

General principle of automatic control of complex systems

In automatic control systems, when detecting a fault, the method of monitoring the passage of a signal is usually used [1, 2]. Signals are applied to the input of the device being tested, usually coinciding with the real input signals. At test points, the parameters of the device’s response to the input signals are measured. If the information obtained at the test points is unsuitable for direct use in evaluating the operation of the device, then it passes through a converter. Usually the information is converted into digital form, since comparison circuits and logic circuits, as a rule, are digital.

Next, the converted information enters a comparator, i.e., a comparing device, where the obtained information is compared with reference values of the measured quantities.

The comparator can compute the deviation of the measured quantity from the reference and issue the magnitude of the deviation, or it can issue a signal with the assessment “acceptable—not acceptable.”

The signal from the comparator is fed to the indication and recording device. The indicating device records the faulty unit or subunit. The indicating device may also provide for recording the degree of deviation of the monitored quantity from the nominal value. This makes it possible to predict the failure of one unit or another.

The time sequence of tests corresponding to a given program is set by the program device. The program device sends commands to the signal generators, controls the input- and output-signal switches, and ensures that the response signal is compared with the reference signal in the comparator.

If a fault signal appears at the comparator output, the program device stops the check, gives an audible fault signal, and switches on the fault-search subprogram.

For monitoring complex devices, the program is entered from an external storage medium. In this case the monitoring program can be changed quickly. A plugboard, punched cards, punched tape, magnetic tape, or magnetic drums may be used as the external storage medium. In specialized self-checking systems the test program is permanent. This type of program is used when there are logical connections among the results of the tests. To increase the reliability of the automatic monitoring system (AMS), the possibility of self-checking may be provided, carried out according to a prescribed subprogram.

Fault-Search Methods

It is known that the major part of forced equipment downtime is the time spent locating faults. Therefore, for complex electronic devices, the development of an optimal fault-search program is of great importance. There are several evaluation criteria, indicated in [2—8], for choosing the optimal route for locating faults.

Thus, Brule, Johnson, and Kletsky, based on information theory, developed a fault-search method characterized by the efficiency of test \(F_k\), equal to the reduction of the uncertainty \(H(P)\) in relation to the cost \(C_k\) of the corresponding test stage:

\[ F_k = \frac{H(P)}{C_k} = \frac{-P \log_2 P - (1-P)\log_2(1-P)}{C_k}, \]

where \(P\) is the a priori probability of a positive test result.

The stages with the largest \(F_k\) are found successively. At each test, the costs of the remaining tests are recalculated, taking into account the tests already performed, and the positive result of each remaining test is recalculated with allowance for the result of the test performed. To use this method in practice, it is necessary to have information on the corresponding costs and probabilities.

The accumulation of these data is an extremely difficult task, especially with regard to the cost of performing elementary operations, which are used to calculate the cost of tests.

The second method is called the “midpoint” method. Its essence is that each subsequent test is made in the middle of the remaining part of the circuit. This method of locating faults

it is advisable to apply it to systems that have sequential signal paths, and when the probabilities of various faults and the time required to locate these faults are identical or unknown. The average time for finding any fault in a particular chain is the same with this method.

The third method, “time–probability,” is based on computing the ratios

\[ \frac{t_i}{P_i}\quad (i=1,2,\ldots,N), \]

where \(t_i\) is the time required to find the fault,
\(P_i\) is the probability of failure of each unit.

On the basis of the computed values \(\frac{t_i}{P_i}\), a sequence of increasing values of the ratio is formed:

\[ \frac{t_{s_1}}{P_{s_1}} < \frac{t_{s_2}}{P_{s_2}} < \cdots < \frac{t_{s_N}}{P_{s_N}}. \]

The search for failures proceeds in the same sequence through the units \(S_1, S_2, \ldots, S_N\).

The advantage of this method is that, when automatic monitoring equipment is used with a semi-fixed search program which, after the beginning of its use, is no longer changed, a comparatively fast and simple programming procedure is possible, provided data are available on the probability of failure and the search time. This method is considered more effective when the probabilities of failures and the times for finding faults are different, and when there is no interdependence among individual faults.

In the case where there are interconnections between the individual parts of the system being checked, while the time costs for searching and the probabilities of faults are known and are not equal, the so-called combined method proves optimal. It is based on the “midpoint” method, taking into account corrections based on data concerning the probabilities of individual faults and the time expenditures for checking individual units of the system.

Automatic monitoring and detection of fault locations in digital computers

Means for monitoring digital computers are subdivided into means included in the error-detection system and means included in the fault-search system.

The means included in the error-detection system issue an error signal without indicating the faulty element. The error signal activates the fault-search system. The search system must effectively carry out the search for the fault, indicating the replaceable unit or module. At present, in most digital computers, fault search is performed with the active intervention of an operator, which sharply reduces the effectiveness of automatic monitoring systems and, moreover, errors may be made by the operator; i.e., the accuracy of detection is reduced, especially when the operator is insufficiently qualified. It should be noted that it is not always possible to divide devices into an automatic monitoring system and a fault-search system.

There are a large number of methods for monitoring the operation of digital computers and for searching for faults. Let us consider the most frequently encountered ones, described in [9—16].

1. Method of repeated computations. When this method is used, a double computation is performed and the results are compared. In solving large problems, the results of individual parts are checked. If, upon comparison, the results of the double computation do not coincide, the solution is repeated. The same problem may be solved on two machines, with intermediate results compared. If the results do not coincide, a double computation may be provided for. If the results fail to coincide twice, a stop may be provided for, followed by switching on the fault-detection system.

If the results of the two solutions coincide, the transition is made to solving the next part. This method of checking is not efficient, since its implementation requires either expensive equipment (two digital computers) or twice the useful machine time. But it is sufficiently reliable, since redundancy of information is achieved by repeating the computations.

2. Check by parity of ones. A binary code always contains an even or odd number of ones. With an even number of ones, the binary code may be denoted by the symbol “1,” and with an odd number by the symbol “0,” or vice versa. To check computations, these symbols are written into check digits.

Let us show by an example how checking is carried out by this method:

When performing arithmetic operations, the presence of binary carries must be taken into account; otherwise the check sum will be incorrect. This checking method detects a single error. When a large number of errors occurs, it is completely unsuitable.

In [12] the advisability of applying this checking method for sequential-action machines is indicated. For parallel-action machines, this method requires a large amount of additional equipment and time to carry out the check.

3. Check codes. The essence of this method is that, in order to detect an error, a code is needed such that, in order to pass from one number represented by this code to the representation by the same code of another number, at least two changes in the digits of the number would be required. The “two-out-of-five” code is widely known and sufficiently described in the literature, as are codes constructed according to the principle of divisibility by some number, for example, a code divisible by 3 or 7. Checking by this method makes it possible not only to detect errors, but also to correct them. However, when this checking method is implemented, the number of digits of the number increases sharply, the logic of the machine becomes substantially more complicated, and much additional equipment is required.

4. Check by modulus. The method being described is based on the idea of using the identity

\[ A = a \mod b. \]

As check symbols in this method, remainders \(a\) of numbers with respect to some fixed modulus \(b\) are used. By performing on the check symbols the same operations as on the numbers, extracting from the result the check symbols for a specified modulus and comparing these results, we can monitor the course of the computations.

This method can be used for checking an arithmetic device, but for checking the control unit this method is unacceptable. Moreover, the use of this method requires additional equipment.

5. Test checking. Test checking is the most widespread method of checking digital computers.

By a test we shall mean the process of specifying some set of input values and observing the outputs of the machine being checked.

Depending on their purpose, tests are subdivided into adjustment, verification, diagnostic, and control test problems.

Adjustment tests are used in the process of adjustment during installation of a machine in order to detect gross errors in the wiring and logic of the machine.

The adjustment of the machine is completed by running previously developed verification tests. These provide for more complete checking of the machine’s units under various operating modes. They take fuller account of the logical circuit of the machine’s units. Control test problems are used for periodic checking during the machine’s normal operation.

Diagnostic test checking is preventive, i.e., it can be carried out during the period of preparation for solving a problem. A diagnostic test is compiled with allowance for the failure rate of each type of element and the approximate time needed to locate them.

Preventive checking consists in solving problems whose answers are known. In this case the machine elements are placed in the most difficult operating modes by changing the supply voltage within permissible limits. For each device of the computer there is its own verification test. Tests of individual devices are combined into a single comprehensive test, which determines the sequence in which the individual tests are to be performed and the duration of their performance. A fault is detected when a certain minimum of the computer’s devices is operable. To indicate precisely the location of the fault under test checking, operator intervention is required.

The literature describes a number of methods of automatic program checking and fault location. Thus, for example, one method of automatic fault-location checking consists in entering a diagnostic program into the machine. The diagnostic program specifies various examples and checks the correctness of their execution. If an example is performed incorrectly, an error-search subprogram is invoked. With its aid, additional examples are executed that make it possible to locate the error. The sequence of checks proceeds from simpler units to more complex ones. The subprogram is constructed so that the elements covered when one example is solved are, as far as possible, excluded when other examples are solved. As soon as a faulty element is detected with the aid of the subprogram, the machine stops, and the indicator device displays a list of elements because of which the error could have occurred. The operator detects the faulty element by the method of successive replacement.

A tester may be used for checking a digital computer. The tester may be a specialized machine with program control. On the basis of a coded description of the logic circuit of the machine being checked, a program is compiled for the computer. Having executed this program, the computer provides a procedure for checking the digital computer being tested. According to

according to the selected procedure, signals from the tester are fed to the corresponding inputs of the machine being checked. Signals from the outputs of the machine being checked are compared with reference signals. Depending on the result of the comparison, a conditional transition is carried out. In other words, the machine is checked by introducing tests into it. Each subsequent test is selected with due regard for the result of the preceding test. When an error is detected, large groups of elements are divided into smaller ones in which the fault may be located. Ultimately, with this testing procedure it is indicated which element has failed.

In the process of solving a problem, monitoring of the progress of program execution is carried out by printing certain intermediate results and sections of the program (addresses and commands). Such a method of monitoring does not provide an idea of the execution of the program as a whole.

In [16] a method is described for program monitoring of the progress of a program. Its idea is that the flowchart of the program is drawn in graphical form according to the sequence in which the problem is solved. One indicator lamp is provided for each block. From the total number of commands included in the individual blocks of the program, a so-called “characteristic” command is selected. Execution of the “characteristic” commands is recorded by the indicating device. If failures occur in the sequence of program execution, the machine stops. The “stop” signal switches all flip-flops controlling the indication into the storage mode. As a result, the lamp that remains lit is the one that preceded the stop. In this way it is established in which block of the program the failure occurred.

This method of monitoring has been applied in the single-address parallel-action machine PERM. To implement this method of monitoring, additional equipment is required.

6. Hardware-logic monitoring. In [12] the essence of hardware-logic monitoring is set forth as follows: “...all the equipment to be monitored is divided into definite groups; for each equipment group its own checking procedure and its own monitoring circuit are developed, which are subsequently coordinated with the operation of the common monitoring device.”

It is possible to develop such monitoring circuits as would indicate the degree of deterioration of the signal parameters and predict in advance the failure of a machine element. In [12, 13] circuits are given for monitoring receiving-transmitting registers, counters, adders, command sampling, and others.

But all these circuits either require a large amount of additional equipment, or are ineffective when a large number of errors appears, or indicate with insufficient accuracy which element has failed.

In almost all digital computers currently being produced in the USSR, circuit monitoring is not provided. Therefore, when organizing circuit monitoring, difficulties arise in connecting additional equipment to the digital computer.

It is probably more expedient to use built-in modules for monitoring some signal parameter, for example, amplitude. The modules should contain a minimum number of radio components so that they can be mounted in the main module of the digital computer. This method of monitoring can be effective when used together with test monitoring. However, very little has yet been done in this direction. Analysis of the materials available in the published literature shows that the problem of automating fault monitoring of electronic

Insufficient attention is still being paid to computers. There are no theoretical developments on this important question, no systematic accumulation and analysis of data on computer operation is being carried out, and built-in self-monitoring systems, etc., are still not provided for in new computer designs.

On the other hand, the speed of computers is growing rapidly, while the complexity of the structure of the machines themselves is steadily increasing, and the economic losses associated with finding and eliminating faults are becoming ever more significant. The use of computers in systems for automatic regulation and control of production processes imposes still stricter requirements with regard to ensuring high reliability and efficiency, and the maintainability of digital computing devices.

The existing individual works on the problem under consideration indicate that these developments are still at an initial stage. It is necessary substantially to intensify work in this direction and, already in machine designs now being developed, to provide for appropriate methods of circuit and test self-monitoring in order to ensure a minimum expenditure of time on locating and eliminating faults during operation.

REFERENCES

1. G. V. Novotny. Automatic control devices for modern complex systems. Elektronika, 35, No. 28, 14—23, 1962.

2. Collection of articles, Automatic Testing of Aircraft and Rocket Equipment. Foreign Literature Publishing House, Moscow, 1962.

3. Brsley, Johnson, Kletsky. Locating faults in technical devices. Zarubezhnaya radioelektronika, No. 7, 1961.

4. Optimal method for locating faults. Zarubezhnaya radioelektronika, No. 3, 18—23, 1959.

5. E. J. Kletsky. Application of information theory to locating faults in technical devices. Zarubezhnaya radioelektronika, No. 9, 124—137, 1961.

6. Winter. Optimal methods for locating faults. Zarubezhnaya radioelektronika, No. 1, 118—126, 1962.

7. Firstman, Glass. Optimal search routes in automatic fault location. Zarubezhnaya radioelektronika, No. 6, 115—124, 1963.

8. I. M. Sindeev. On the question of synthesizing logic circuits for fault location and state monitoring of complex systems. Technical Cybernetics. Publishing House of the Academy of Sciences of the USSR, No. 2, 1963.

9. S. V. Peterson. On adder control. Cybernetic Collection, No. 4. Foreign Literature Publishing House, Moscow, 1962.

10. Systems of automatic control and fault location in radio-electronic equipment (review). Zarubezhnaya radioelektronika, No. 5, 108—122, 1961.

11. F. I. Belov, F. S. Soloveichik. Problems of the Reliability of Radio-Electronic Apparatus. Moscow—Leningrad, 1961.

12. Collection of articles, Computing Technology, issue 4, Moscow, 1962.

13. E. I. Klyamko. Circuit and Test Control of Automatic Digital Computers. Moscow, 1963.

14. S. Seshu, D. N. Freeman. Diagnostics of asynchronous sequential switching systems. Ekspress-informatsiya, vychislitel’naya tekhnika, No. 8, 1963.

15. Nonken Howard R. Missile computer has “self—checking” capability. Electron. Inds., 20, No. 7, 100—102, 1961.

16. Tsui Frank F. A flexible and inexpensive method of monitoring program execution in a digital computer. IRE Trans. Electronic comput., 10, No. 2, 253—259, 1961.