PHYSICAL CHEMISTRY

Yu. S. VARSHAVSKII, A. B. SHEININ

ON THE ENTROPY OF SYSTEMS CONTAINING COMPONENTS THAT ARE DIFFICULT TO DISTINGUISH

(Presented by Academician A. A. Grinberg, 23 VII 1962)

The macroscopic description of a physical system contains essentially incomplete information concerning its detailed properties, which are included in the concept of the “microstate of the system.” The measure of the information absent from the macroscopic description of a system is its entropy. In the cases usually considered, the lack of information reduces to uncertainty in the coordinates, momenta, and internal states of the particles composing the system. For multicomponent systems an uncertainty of yet another type is characteristic, namely, uncertainty as to the belonging of a given particle to one component or another. This additional uncertainty corresponds to the well-known logarithmic term in the expression for the entropy. In the case where the system consists of components with very similar properties, certain new relations arise, leading to a change in the magnitude of the logarithmic term. These relations are considered below in connection with the particular question of the entropy of mixing of ideal gases.

As is known, the formation of 1 mole of a binary ideal gas mixture from equal quantities of the components at constant temperature and pressure is accompanied by an increase in entropy of magnitude \(\Delta S = R \ln 2\), independent of the nature of the gases being mixed. The mixing of two portions of one and the same gas does not lead to an increase in entropy: \(\Delta S = 0\). Thus, the emergence of the slightest distinction between the gases being mixed abruptly increases the entropy of mixing from 0 to \(R \ln 2\), while a further increase in the degree of distinction has no effect on this quantity. This fact constitutes the content of the so-called Gibbs paradox.

A detailed consideration of the question, based on the conception of entropy as a measure of the lack of information about the microstate of a system, leads, however, to substantially different results. In accordance with this conception, the entropy of a thermodynamic system is proportional to that amount of information which can be obtained as a result of a detailed investigation of all the particles composing the system. In considering the question of the change in entropy in some process, not all properties of the particles are essential, but only those of them whose information changes as a result of the course of the process. In the case of mixing ideal gases, such a property is the belonging of a particle to one component or another. In this case, investigation of the system at the molecular level can be reduced to the recognition of all the particles composing the system. The aggregate of means by which such an investigation could be carried out will be called a “recognizing device.” We shall denote the amount of information obtained with the aid of the recognizing device by \(I_0\) or \(I\), depending on whether it is a question of the initial (before mixing) or final (after mixing) state of the system. The magnitude of the entropy of mixing \(\Delta S\) will then be determined by the obvious expression \(\Delta S = a(I - I_0)\), where \(a\) is a proportionality coefficient. If \(I_0\) and \(I\) are measured in binary units (bits), and \(\Delta S\) in thermodynamic entropy units, then, as is known ((1), p. 22), \(a = k \ln 2\) (\(k\) is Boltzmann’s constant). Thus,

\[ \Delta S = k \ln 2 (I - I_0). \]

The process we are considering consists in mixing equal quantities (0.5 mole each) of pure substances A and B, which have the properties of an ideal gas and initially occupy two parts of a certain vessel that are equal in volume. In the initial state the gases are separated by a partition, and the nature of each particle is completely determined by the part of the vessel in which this particle is located. Regardless of the degree of difference between A and B, the use of a recognizing device would not in this case lead to the acquisition of any information: \(I_0 = 0\), and, consequently,

\[ \Delta S = I k \ln 2 . \tag{2} \]

Equation (2) makes it possible to reduce the calculation of the entropy of mixing to the determination of the amount of information that can be obtained as a result of applying a recognizing device to the study of the mixture formed. Since the quantity \(I\) depends on the degree of difference between A and B, it is expedient to consider separately three cases.

1. Gases A and B are identical. In this case, obviously, the recognizing device is incapable of distinguishing particles A and B, and the amount of information \(I\) that can be obtained with its aid is equal to zero. Consequently, in accordance with (2), for this case

\[ \Delta S = 0 . \tag{3} \]

1. Gases A and B differ from one another so much that the recognizing device distinguishes their particles without error. Since the mixture is equimolecular, the a priori probability that a certain particle belongs to one of the components is equal to \(1/2\). Error-free recognition of each particle is associated with the receipt of one bit of information; as a result of studying all \(N\) particles constituting the system, \(N\) bits will be obtained (\(N\) is Avogadro’s number). Thus, \(I = N\); since \(kN = R\), it follows from (2) that in this case

\[ \Delta S = R \ln 2 . \tag{4} \]

1. Gases A and B are different, but difficult to distinguish, i.e., are so similar to one another that during operation of the recognizing device there is a nonzero probability \(w\) of error, in which a particle A is identified as B or vice versa. In this case the procedure of recognizing the particle does not completely remove the uncertainty as to its belonging to one or the other component. For example, if a molecule is recognized as A, then the probability that it belongs to A is \(1 - w\), and the probability that it belongs to B is \(w\). The uncertainty of this situation is measured by the quantity \(h\), known in information theory as entropy (“information entropy”), \(h = [w \log_2 w + (1 - w)\log_2(1 - w)]\) bits (Shannon’s formula). The amount of information obtained in each act of recognition is therefore equal to \(1 - h = 1 + w\log_2 w + (1 - w)\log_2(1 - w)\). The amount of information obtained in studying all \(N\) molecules will obviously be \(N[1 + w\log_2 w + (1 - w)\log_2(1 - w)]\), whence, according to equation (2),

\[ \Delta S = R \ln 2 \,[1 + w \log_2 w + (1 - w)\log_2(1 - w)] . \tag{5} \]

Equation (5) is quite general for the process under consideration. The trivial results expressed by equations (3) and (4) are easily obtained from (5) as limiting cases corresponding to \(w = 1/2\) (identical or indistinguishable gases) and \(w = 0\) (well-distinguishable gases). It is essential that (5) provides a continuous transition between these limiting cases, thereby eliminating the difficulties associated with the Gibbs paradox.

The operation of a recognizing device of any type must obviously be based on the measurement of some properties characterizing particles A and B. The magnitude \(w\) (and consequently also \(\Delta S\)) depends on the degree of difference between A and B with respect to the properties used in recognition, and also-

also on the accuracy of measurement of these properties. As an illustration, let us calculate the quantity \(\Delta S\) as a function of the degree of distinguishability of the particles of the gases being mixed, for one typical case.

Let the identification of particles be carried out on the basis of measuring some property \(X\), whose values for molecules A and B are equal to \(X_A\) and \(X_B\), respectively. For definiteness let \(X_B > X_A\), and denote \(X_B - X_A = 2\Delta\). We shall assume that the errors in measuring \(X\) obey the normal distribution law with variance \(\sigma^2\). Obviously, a molecule A will be erroneously recognized as B in the case where the deviation of the measured quantity \(X\) from the true value \(X_A\) exceeds \(+\Delta\). The probability of this event is

\[ w = \frac{1}{2}\left[1-\Phi\left(\frac{\Delta}{\sigma}\right)\right], \]

where

\[ \Phi\left(\frac{\Delta}{\sigma}\right) = \frac{2}{\sqrt{2\pi}} \int_{0}^{\Delta/\sigma} e^{-\Delta^{2}/2\sigma^{2}}\,d\left(\frac{\Delta}{\sigma}\right). \]

Fig. 1. Dependence of the entropy of mixing of ideal gases on the degree of distinguishability of their molecules

From this expression for \(w\), with the aid of tables of the probability integral, the quantity \(\Delta S\) can be calculated from equation (5) for various \(\Delta/\sigma\). The results of such a calculation were used to construct the graph (see Fig. 1), expressing the dependence of the entropy of mixing on the degree of distinguishability of particles A and B, characterized by the ratio \(\Delta/\sigma\). Under the conditions \(\Delta \ll \sigma\) and \(\Delta \gg 4\sigma\), the equalities \(\Delta S = 0\) and \(\Delta S = R\ln 2\), corresponding to the two limiting cases considered in the classical theory, are satisfied with a high degree of accuracy.

The fact that the magnitude of the entropy of mixing is determined by the possibility of recognizing the molecules of the gases being mixed as different was noted already by Gibbs. Considering the case of mixing identical gases, Gibbs wrote ((\(^{2}\), p. 166) that the energy and entropy of these gas masses after mixing remain the same as before these masses were mixed, because we see no difference whatever in the substance of these two masses. The traditional point of view recognized only two gradations of this possibility: either we are completely unable to distinguish the gases being mixed, or we distinguish them without error. The Gibbs paradox turns out to be a trivial consequence of this postulated discontinuity in the degree of distinguishability of gases*.

Let us note that the result expressed by equation (5) can also be obtained by the usual means of statistical thermodynamics, without using the concepts of information theory. The identity of the results obtained by both methods emphasizes the equivalence of “physical” and “informational” entropy and once again testifies to the legitimacy of discussing thermodynamic questions in terms of information theory.

The authors consider it their pleasant duty to express their gratitude to Prof. L. E. Gurevich for a useful discussion.

Received
20 VII 1962

REFERENCES

1. L. Brillouin, Science and Information Theory, Moscow, 1960.
2. J. Willard Gibbs, Collected Works, 1, Thermodynamics, N. Y.—London—Toronto, 1928.
3. R. M. Noyes, J. Chem. Phys., 34, No. 6, 1983 (1961).

* Precisely this view of the origin of the Gibbs paradox is expressed by R. M. Noyes (\(^{3}\)) in connection with the question of the entropy of mixing of gases that mutually transform into one another.