UDC 517.946

ON A BOUNDARY VALUE PROBLEM

FOR AN EQUATION OF MIXED TYPE

WITH TWO LINES OF PARABOLIC DEGENERATION*)

A. M. NAKHUSHEV

In this paper, as a model equation of mixed elliptic-hyperbolic type with two nonintersecting lines of parabolic degeneration, the equation

\[ y(y-1)u_{xx}+u_{yy}=0 \tag{L} \]

is considered in a domain containing within it intervals of both straight lines \(y=0\), \(y=1\).

§ 1. Introduce the following notation:

\[ \omega(y)=1/4\,k'\sqrt{-k}+1/8\arcsin k', \]

\[ \tilde y(y)=-(3/2a)^{2/3}(\omega\mp\pi/16)^{2/3}, \tag{1.1} \]

\[ 0\leq y\leq 1, \]

\[ \omega(y)=1/4\,k'\sqrt{k}-1/8\ln\left|k'+2\sqrt{k}\right|, \qquad \tilde y(y)=(3\omega/2a)^{2/3}, \tag{1.2} \]

\[ y\leq 0,\quad y\geq 1;\qquad k(y)\equiv y(y-1); \]

\[ a=\mathrm{const},\qquad \tilde x=x/a,\qquad \pi<8a<2\pi; \tag{1.3} \]

\[ a\,\xi_1(x,y)=x+\omega-\pi/16,\qquad a\,\eta_1(x,y)=x-\omega+\pi/16; \tag{1.4} \]

\[ b(\tilde y)=[\tilde y'(y)]^{-2}\tilde y''(y), \qquad \tilde\omega(y)=\exp\left(\frac12\int_0^{\tilde y} b(s)\,ds\right). \tag{1.5} \]

In the plane of the variables \(x,y\), consider the points

\[ A_1(0,0),\quad A_0(\pi/8,0),\quad A_2(a,0),\quad B(a-\pi/16,1/2),\quad A_3(a,1), \]

\[ A_4(0,1),\quad A(a-\pi/8,1),\quad B_1(\pi/16,1/2),\quad A_5(a-\pi/8,0). \]

Let \(D^*\) be a finite simply connected domain bounded by: 1) the arc \(\sigma_1\):
\((x-a/2)^2+\omega^2=a^2/4,\ y\geq 1\); 2) the arc \(\sigma_0\):
\((x-a/2)^2+\omega^2=a^2/4,\ y\leq 0\); by the characteristics \(A_1B_1\), \(\eta_1=\pi/8a\), \(B_1A_4\): \(\xi_1=0\), \(A_2B\): \(\xi_1=1-\pi/8a\), \(BA_3\): \(\eta_1=1\) of equation \((L)\). Further, let \(B_0\) \((D_0)\) be the point of intersection of the characteristics \(A_0B_0\): \(\eta_1=\pi/4a\), and \(AB_0\): \(\xi_1=1-\pi/8a\) \((AD_0: \eta_1=\pi/8a\) and \(B_1A_4)\); \(D^{-}\) be the part of the domain \(D^*\) lying above the curve \(D_0AB\); \(D^{+}\) the elliptic part of the domain \(D\); \(\Delta\) \((\Delta^*)\) the hyperbolic part \(D\), lying above the curve \(A_4D_0A\) \((ABA_3)\).

*) The work is essentially a detailed exposition of the article [1].

Problem 1. Find a function \(u(x,y)\) with the following properties:
1) \(u\in C(D^*-A_0-A_2)\); at the point \(A_0\) \((A_2)\) \(u\) tends (may tend) to \(\infty\) of order \(\alpha<1/6\) (of logarithmic order); 2) \(u_y\,(u_x)\) is continuous everywhere in the closed domain \(\overline{D^*}\), except for \(A_0\); \(u\), possibly, \(A, A_i\ (i=1,2,\ldots,5)\) and the characteristics issuing from them, where it tends to \(\infty\) of order \(<5/6\) \((<7/6)\) and \(<1\) \((<1)\), respectively; 3) \(u\) is a regular solution of equation \((L)\) everywhere in \(D^*\), except, possibly, for the characteristics issuing from the points \(A_0, A, A_5\); 4) \(u\) satisfies the boundary conditions:

\[ (\widetilde{\omega}u)_{\widetilde{\sigma}_1}=\varphi_1(\widetilde{x}),\qquad 0\leq \widetilde{x}\leq 1; \tag{1.6} \]

\[ (\widetilde{\omega}u)_{A_4D_0}=\psi_1(\eta_1),\qquad 0\leq \eta_1\leq x_0; \tag{1.7} \]

\[ (\widetilde{\omega}u)_{A_3B}=\psi_2(\xi_1),\qquad x_0\leq \xi_1\leq 1; \tag{1.8} \]

\[ (\widetilde{\omega}u)_{BB_0}=\psi_3(\eta_1),\qquad 1\leq \eta_1\leq \pi/4a; \tag{1.9} \]

\[ (\widetilde{\omega}u)_{\sigma_0}=\varphi_0(\widetilde{x}),\qquad 0\leq \widetilde{x}<1, \tag{1.10} \]

where \(\widetilde{\omega}\) is defined by formula (1.5), and in (1.1) before \(\pi/16\) the minus sign is taken, and \(x_0=1-\pi/8a\).

The following assumptions are made concerning the functions \(\varphi_i,\psi_i\):

\[ \varphi_1(\widetilde{x})\in C^2(0<\widetilde{x}<1),\qquad \varphi_1(\widetilde{x})=O(1)(\widetilde{x}-\widetilde{x}^{\,2})^{\chi_1}; \tag{1.11} \]

\[ \psi_1(\eta_1)\in C^7(0<\eta_1\leq x_0),\qquad \psi_1^{(i)}(\eta_1)=O(1)\eta_1^{\chi_2-i},\quad i=0,1; \tag{1.12} \]

\[ \psi_2(\xi_1)\in C^7(x_0\leq \xi_1<1),\qquad \psi_2^{(i)}(\xi_1)=O(1)(1-\xi_1)^{\chi_2-i}; \tag{1.13} \]

\[ \psi_3(\eta_1)\in C(1\leq \eta_1\leq \pi/4a),\qquad \psi_3(\eta_1)\in C^4(1<\eta_1<\pi/4a); \tag{1.14} \]

\[ 1<\chi_i=\mathrm{const},\qquad \psi_3'(\eta_1)=O(1)(\pi/4a-\eta_1)^{-\alpha-5/6}; \tag{1.15} \]

\(\psi_3'(\eta_1)\) at \(\eta_1=1\) may tend to \(\infty\) of order \(\leq 2/3\); \(\varphi_0(\widetilde{x})\in C(0\leq \widetilde{x}<1)\), \(\varphi_0(\widetilde{x})\in C^2(0<\widetilde{x}<1)\), and \(\varphi_0\) at the point \(\widetilde{x}=1\) tends, generally speaking, to \(\infty\) of logarithmic order.

The uniqueness of the solution of Problem 1 is proved in exactly the same way as in the case of Problem \(A\), considered in [2].

§ 2. 1. In equation \((L)\) we pass to the variables \(\widetilde{x}=x/a\), \(\widetilde{y}=(3\omega/2a)^{2/3}\) for \(y>1\), \(\widetilde{y}=-(3/2a)^{2/3}(\pi/16-\omega)^{2/3}\) for \(0<y<1\), and to the function \(z=\widetilde{\omega}u\). Then \(z\) will satisfy the equation

\[ \widetilde{y}z_{\widetilde{x}\widetilde{x}}+z_{\widetilde{y}\widetilde{y}}+c(\widetilde{y})z=0, \tag{\(\widetilde{L}\)} \]

where \(-4c(\widetilde{y})=2b_y+b^2\) admits derivatives of any order, and \(c(0)>0\) (see [2, 3]).

If \(M\) is a certain set in the plane of the variables \(x,y\), then its image in the plane of the variables \(\widetilde{x},\widetilde{y}\) will henceforth be denoted by \(\widetilde{M}\). Obviously, \(\widetilde{D}^{+}\) is a finite simply connected domain bounded by the normal contour

\[ \widetilde{\sigma}_1:\quad (\widetilde{x}-1/2)^2+\frac{4}{9}\widetilde{y}^{\,3}=1/4,\quad \widetilde{y}\geq 0 \tag{2.1} \]

of equation \((\widetilde{L})\) and by the segment \(\widetilde{A}_4\widetilde{A}\) of the straight line \(\widetilde{y}=0\).

Problem 2 (Holmgren). Find a solution of equation \((\tilde L)\), regular in the domain \(\tilde D^+\), satisfying the boundary conditions (1.6)

\[ z=\varphi_1(\tilde x),\quad (\tilde x,\tilde y)\in\tilde\sigma_1,\quad 0\leqslant \tilde x\leqslant 1; \]

and

\[ z_{\tilde y}=\nu(\tilde x),\quad \tilde y=0,\quad 0<\tilde x<1,\quad \tilde x\ne x_0, \tag{2.2} \]

where \(\nu(\tilde x)\in C^1(0<\tilde x<1,\ \tilde x\ne x_0)\) and, for \(\tilde x=0,x_0,1\), may tend to \(\infty\) of order \(<2/3\).

For convenience, in equation \((\tilde L)\) we pass to new variables:

\[ x=2\tilde x-1,\quad y=2^{2/3}\tilde y. \tag{2.3} \]

Then \((\tilde L)\), (1.6), and (2.2) take the following form:

\[ Tz=yz_{xx}+z_{yy}=-4^{-2/3}c(4^{-1/3}y)z=c_0(y)z; \tag{2.4} \]

\[ z=\varphi_1((x+1)/2)=\tilde\varphi(x),\quad (x,y)\in\sigma,\quad |x|\leqslant 1; \tag{2.5} \]

\[ z_y=4^{-1/3}\nu((x+1)/2)=\tilde\nu(x),\quad y=0,\quad |x|<1,\quad x\ne 2x_0-1. \tag{2.6} \]

The transformation (2.3) carries \(\tilde D^+\) into the domain \(\Omega\), bounded by the curve \(\sigma\):

\[ x^2+\frac{4}{9}y^3=1 \]

and the segment \(|x|\leqslant 1,\ y=0\).

The Green’s function of Problem 2 in the domain \(\Omega\) is given by formula [4]

\[ G(x,y;\xi,\eta)=u(x,y;\xi,\eta)+v(x,y;\xi,\eta), \tag{2.7} \]

where

\[ u=\gamma_1(r_1^2)^{-1/6}\mathfrak F\left(\frac16,\frac16,\frac13;\ 1-r^2/r_1^2\right), \]

\[ v=-(R^2)^{-1/6}u(x,y;\bar\xi,\bar\eta), \tag{2.8} \]

\[ \left. \begin{array}{c} r^2\\ r_1^2 \end{array} \right\} =(x-\xi)^2+\frac49\left(y^{3/2}\mp\eta^{3/2}\right)^2,\quad R^2=\xi^2+\frac49\eta^3, \tag{2.9} \]

\[ \bar\xi=\xi/R^2,\quad \bar\eta^{3/2}=\eta^{3/2}/R^2,\quad 4\pi\Gamma(1/3)\gamma_1=(4/3)^{1/3}\Gamma^2(1/6). \tag{2.10} \]

In [2] it is proved that there exists a unique solution \(z\) of Problem 2, and it can be represented in the form

\[ z=z_0(x,y)+Z(x,y)+\int_{-1}^{1}\tilde\nu(t)N(t,x,y)\,dt, \tag{2.11} \]

where

\[ z_0=N_0+F=-\int_{-1}^{1}\tilde\nu(\xi)G(\xi,0;x,y)\,d\xi+ \]

\[ +\int_{-1}^{1}\tilde\varphi(\xi)A_\xi\left[\overline{G(\xi,\eta;x,y)}\right]\,d\xi, \tag{2.12} \]

\[ A_\xi\equiv \eta(d\eta/d\xi)\partial/\partial\xi-\partial/\partial\eta, \]

and the bar over \((\xi,\eta)\) means that \((\xi,\eta)\in\sigma\); \(Z\) and \(N\), respectively, are solutions of the following unconditionally solvable Fredholm equations:

\[ Z(x,y)+\int_{\Omega} c_0(\eta)G(\xi,\eta;x,y)Z(\xi,\eta)\,d\xi\,d\eta=Z_0(x,y), \tag{2.13} \]

\[ N(t,x,y)+\int_{\Omega} c_0(\eta)G(\xi,\eta;x,y)\times \]

\[ \times N(t,\xi,\eta)\,d\xi\,d\eta=N_0(t,x,y), \tag{2.14} \]

\[ Z_0=-\int_{\Omega} c_0(\eta)G(\xi,\eta;x,y)F\,d\xi\,d\eta, \]

\[ N_0=\int_{\Omega} c_0(\eta)GG(t,0;\xi,\eta)\,d\xi\,d\eta. \tag{2.15} \]

1. From (2.15) we have

\[ N_0(y,x,0)=\mathfrak{N}(x,y)=\int_{\Omega} c_0(\eta)\times \]

\[ \times G(\xi,\eta;x,0)G(y,0;\xi,\eta)\,d\xi\,d\eta. \tag{2.16} \]

Below it will be shown that the function \(\mathfrak{N}(x,y)\in C^\infty(-1<x,\ y<1,\ x\ne y)\), while for \(x=y\) the first derivatives have a logarithmic singularity, which can be isolated explicitly.

Consider the function

\[ \mathfrak{N}_i(x,y)=\int_{\Omega}\eta^i G(\xi,\eta;x,0)G(y,0;\xi,\eta)\,d\xi\,d\eta,\quad i\ge 0. \tag{2.17} \]

From (2.7)—(2.10) it is easy to see that

\[ G(\xi,\eta;x,0)=\gamma_1(R_x^2)^{-1/6}-\gamma_1(\bar R_x^2)^{-1/6}, \tag{2.18} \]

where

\[ R_x^2=(x-\xi)^2+4/9\,\eta^3,\quad \bar R_x^2=(1-x\xi)^2+4/9\,x^2\eta^3. \]

Consequently,

\[ \mathfrak{N}_i(x,y)=\gamma_1^2\int_{\Omega}\eta^i(R_xR_y)^{-1/3}\,d\xi\,d\eta-\gamma_1^2\int_{\Omega}\eta^i\left[(R_x\bar R_y)^{-1/3}+\right. \]

\[ \left.+(\bar R_xR_y)^{-1/3}-(\bar R_x\bar R_y)^{-1/3}\right]\,d\xi\,d\eta= \]

\[ =\gamma_1^2\mathfrak{N}_i^1(x,y)+\mathfrak{N}_i^2(x,y). \tag{2.19} \]

Obviously, \(\mathfrak{N}_i^2(x,y)\in C^\infty(-1<x,y<1)\) for any \(i\ge 0\). The integral \(\mathfrak{N}_i^1\), after passing to the repeated integral and transforming by means of the substitution \(\eta=(9/4)^{1/3}[(1-\xi^2)(1-t)]^{1/3}\), takes the form

\[ \mathfrak{N}_i^1(x,y)=a_i\int_{-1}^{1}(1-\xi^2)^{i/3}(\zeta\zeta_0)^{1/6}\times \]

\[ \times F_1\left(1,\frac16,\frac16,\frac{i+4}{3},\zeta,\zeta_0\right)\,d\xi, \tag{2.20} \]

where \(0<a_i=\mathrm{const};\quad \zeta=\zeta(x)=\dfrac{1-\xi^2}{1+x^2-2x\xi};\quad \zeta_0=\zeta(y);\quad F_1\) is the hypergeometric function of the variables \(\zeta,\zeta_0;\quad 0\le \zeta,\zeta_0\le 1.\)

The following identity holds (see [4] § 17):

\[ \begin{aligned} &\mathrm{B}(1,\gamma-1)F_1(1,\beta,\beta,\gamma,u,v)+\\ &\quad+\mathrm{B}(1,1+2\beta-\gamma)F_1(1,\beta,\beta,2+2\beta-\gamma,1-u,1-v)+\\ &\quad+\exp(\gamma\pi i)\mathrm{B}(1+2\beta-\gamma,\gamma-1)(-u)^{-\beta}\times\\ &\quad\times(-v)^{-\beta}F_1(1+2\beta-\gamma,\beta,\beta,2\beta,1/u,1/v), \end{aligned} \]

where \(\mathrm{B}\) is the beta-function, \(-\pi<\arg(-u),\arg(-v)<\pi\), \(\operatorname{Im}u,\operatorname{Im}v\leqslant0\). Hence, using the known relation
\((1-v)^{-\alpha}\mathfrak{F}(\alpha,\beta,\beta+\beta',(u-v)/(1-v))=F_1(\alpha,\beta,\beta',\beta+\beta',u,v)\) and the fact that
\(F_1(\alpha,\beta,\beta,\gamma,u,v)=F_1(\alpha,\beta,\beta,\gamma,v,u)\), we obtain

\[ F_1\left(1,\frac{1}{6},\frac{1}{6},\frac{i+4}{3},\xi,\xi_0\right)= \]

\[ \begin{aligned} &=a(i)F_1\left(1,\frac{1}{6},\frac{1}{6},\frac{3-i}{3},1-\xi,1-\xi_0\right)+\\ &\quad+b(i)(\xi\xi_0)^{-1/6}\xi^{-i/3}(1-\xi)^{i/3}\times\\ &\quad\times\mathfrak{F}\left(-\frac{i}{3},\frac{1}{6},\frac{1}{3};\frac{\xi_0-\xi}{\xi_0(1-\xi)}\right), \end{aligned} \tag{2.21} \]

\[ \xi_0\geqslant \xi; \]

\[ F_1\left(1,\frac{1}{6},\frac{1}{6},\frac{i+4}{3},\xi,\xi_0\right)= \]

\[ \begin{aligned} &=a(i)F_1\left(1,\frac{1}{6},\frac{1}{6},\frac{3-i}{3},1-\xi,1-\xi_0\right)+\\ &\quad+b(i)(\xi\xi_0)^{-1/6}\xi_0^{-i/3}(1-\xi_0)^{i/3}\times\\ &\quad\times\mathfrak{F}\left(-\frac{i}{3},\frac{1}{6},\frac{1}{3};\frac{\xi-\xi_0}{\xi(1-\xi_0)}\right), \end{aligned} \tag{2.22} \]

\[ \xi\geqslant \xi_0; \]

where \(a(i)=(1+i)/i,\ b(i)=\Gamma(-i/3)\Gamma(i/3+4/3)/\Gamma(1/3),\ i/3\ne0,1,\ldots\)

Let \(x\leqslant y\). Then from (2.20), (2.21), and (2.22) we have

\[ \begin{aligned} \mathfrak{N}_i(x,y)=\mathscr{L}_i^{*}(x,y) &+m_i\int_{-1}^{(x+y)/2}(y-\xi)^{2i/3}\times\\ &\quad\times\mathfrak{F}\left(-\frac{i}{3},\frac{1}{6},\frac{1}{3}; \frac{(y-x)(y+x-2\xi)}{(y-\xi)^2}\right)d\xi+\\ &+m_i\int_{(x+y)/2}^{1}(\xi-x)^{2i/3}\mathfrak{F}\left(-\frac{i}{3},\frac{1}{6},\frac{1}{3};\right.\\ &\quad\left.\frac{(y-x)(2\xi-x-y)}{(\xi-x)^2}\right)d\xi =\mathscr{L}_i^{*}+I_1+I_2. \end{aligned} \]

Here and in what follows \(\mathscr L_i^{*}\) denotes some function of class \(C^\infty\) \((-1<x,\ y<1)\), and \(m_i\) denotes some nonzero constant. In the integrals \(I_1\) and \(I_2\) we make the substitutions \(\xi=(x-yt)/(1-t)\) and \(\xi=(y-xt)/(1-t)\), respectively. Then their sum \(I\) takes the form

\[ I=m_i(y-x)^{\frac{2i}{3}+1} \left\{ \int_{-1}^{(1+x)/(1+y)} + \int_{-1}^{(1-y)/(1-x)} \right\} \times \]

\[ \times (1-t)^{-\frac{2i}{3}-2} \mathscr F\left(-\frac{i}{3},\frac{1}{6},\frac{1}{3};1-t^2\right)\,dt . \]

It is now not hard to see that

\[ I=m_i(y-x)^{\frac{2i}{3}+1} \left\{ \int_{(1+x)/(1+y)}^{1} + \int_{(1-y)/(1-x)}^{1} \right\} \times \]

\[ \times (1-t)^{-\frac{2i}{3}-2}\mathscr F_p(t)\,dt + \]

\[ +\,m_i(y-x)^{2i/3+1}=-J+m_i(y-x)^{2i/3+1}, \]

where \(p>2i/3\),

\[ \mathscr F_p(t)= \mathscr F\left(-\frac{i}{3},\frac{1}{6},\frac{1}{3};1-t^2\right) - \]

\[ -\sum_{k=0}^{p} \frac{(-i/3)_k(1/6)_k}{(1/3)_k\,k!}\,(1-t^2)^k \]

and the symbol \((\lambda)_k\) denotes the quantity \(\Gamma(\lambda+k)/\Gamma(\lambda)\).

After a simple change of the variable of integration in the integrals entering into \(J\), we obtain

\[ J=m_i(1+y)^{1+2i/3} \int_0^1 t^{-2-2i/3}\mathscr F_p(x^*t(2-x^*t))\,dt+ \]

\[ +\,m_i(1-x)^{1+2i/3} \int_0^1 t^{-2-2i/3}\mathscr F_p(y^*t(2-y^*t))\,dt, \]

where \(x^*=(y-x)/(1+y)\), \(y^*=(y-x)/(1-x)\). Therefore \(J\) is a function of type \(\mathscr L_i^{*}\).

Thus it has been proved that, by virtue of (2.19), (2.20), the function \(\mathfrak R_i\) from (2.17) is representable in the form

\[ \mathfrak R_i(x,y)=m_i|y-x|^{1+2i/3}+\mathscr L_i^{*}, \quad i/3\ne 0,1,2,\ldots \tag{2.23} \]

There is no doubt that (2.23) is also valid for \(x>y\). Let now \(i/3=0,1,2,\ldots\). Then

\[ \mathfrak R_i(x,y)=m_i|y-x|^{1+2i/3}\ln|y-x|+\mathscr L_i^{*}. \tag{2.24} \]

For the proof it suffices to extend identities (2.21) and (2.22) to the values of \(i\) under consideration. This is done in the same way as for the hypergeomet-

of functions of one variable [5]. Namely: we write (2.21), (2.22) for \(i=3m+\varepsilon\), where \(m=0,1,2,\ldots\), \(\varepsilon\) is an arbitrarily small positive number, and then pass to the limit as \(\varepsilon\to0\).

Since \(c_0(y)\in C^\infty\left(y>-(3\pi/8a)^{2/3}\right)\), by Taylor’s formula one may write:
\[ c_0(y)=\sum_{i=0}^{q} d_i y^i+O(y^q),\qquad d_0=c_0(0)<0,\quad q\geqslant1. \]
Consequently (see (2.16), (2.17), (2.23), (2.24)),
\[ N_0(t,x,0)=\sum_{i=0}^{q} d_i\mathfrak N_i(x,t)+\mathfrak M_{\bar q}(x,t), \tag{2.25} \]
where \(\mathfrak M_{\bar q}\in C^\infty(-1<x,\ t<1,\ x\ne t)\), \(\mathfrak M_{\bar q}\in \bar C^q(-1<x,\ t<1)\), if \(\bar q=1+2[q/3]\) and \(q/3\ne1,2,3,\ldots\); whereas if \(q/3=1,2,3,\ldots\), then \(\mathfrak M_{\bar q}\in \bar C^{q-1}(-1<x,\ t<1)\), and the \(\bar q\)-th derivative at \(x=t\) may become infinite of order not higher than logarithmic.

1. Obviously, the function \(N_0(t,x,y)\) is a solution of the homogeneous Holmgren problem for the equation
   \[ TN_0+c_0(y)G(t,0;\ x,y)=0,\qquad (x,y)\in\Omega . \]
   From the analytic character of solutions of an elliptic equation with analytic coefficients (see, for example, [6]) it follows unconditionally that \(N_0\in C^\infty(\Omega,\ |t|<1)\). Moreover, it turns out that \(N_0\) has derivatives of arbitrary order with respect to \(x\) and \(t\) when
   \[ (x,y)\in\Omega\cup\{y=0,\ |x|<1,\ x\ne t,\ |t|<1\}. \]
   Indeed, let \(t<t+\varepsilon<x\), and let \(\Omega_1(\Omega_2)\) be a subdomain of the domain \(\Omega\), where \(\xi<t+\varepsilon\) (\(\xi>t+\varepsilon\)). In view of (2.7) and (2.15), it suffices to establish that the function
   \[ U_1(t,x,y)+U_2(t,x,y)= \]
   \[ =\left(\int_{\Omega_1}+\int_{\Omega_2}\right)c_0(\eta)u(\xi,\eta;\ x,y)\,u(t,0;\ \xi,\eta)\,d\xi\,d\eta \]
   has the property indicated above.

It is easy to see that \(U_1\) (\(U_2\)) has derivatives of all orders with respect to \(x\) (\(t\)) for
\[ (x,y)\in\Omega\cup\{y=0,\ |x|<1,\ |t|<1\}, \]
and they may be obtained by direct differentiation under the integral sign. The fact that \(U_1\) (\(U_2\)) admits derivatives with respect to \(t\) (\(x\)) is established by means of the known identity
\[ u_x(\xi,\eta;\ x,y)=-u_\xi(\xi,\eta;\ x,y) \]
and by successive application of the operations of differentiation and integration by parts.

Starting from the integral equation (2.14) and the properties of the function \(N_0\) given here, one can prove (see [4], Ch. 1, § 17, where the case \(0<c_0(y)\equiv\mathrm{const}\) is considered) that the function
\[ N(t,x,0)\in C^\infty(-1<x,\ t<1,\ x\ne t) \]
and is representable in the form (2.25).

1. In (2.11) let us return to the variables \(\tilde x,\tilde y\) by formula (2.3) and, as a result, set \(\tilde y=0,\ \tilde x=x\). Then, according to (2.12), we shall have
   \[ \tau(x)=\gamma_1\int_0^1 \nu(t)\left[L(x,t)-|t-x|^{-1/3}+\right. \]
   \[ \left.+(t+x-2tx)^{-1/3}\right]dt+\Phi(x), \tag{2.26} \]
   where
   \[ \gamma_1L(x,t)=N(2t-1,\ 2x-1,\ 0), \]
   \[ \Phi(x)=F(2x-1,0)+Z(2x-1,0). \]

From (2.12), taking into account that

\[ A_{\xi}[G(\xi,\eta;x,y)] =\gamma_{1}/(2\eta)(1-x^{2}-4/9\,y^{3})(r_{1}^{2})^{-7/6}\times \]

\[ \times \mathcal{F}\left(1/6,\ 7/6,\ 1/3;\ 1-(r/r_{1})^{2}\right), \tag{2.12′} \]

we obtain (see [4, 9])

\[ F(x,0)=(4/9)^{1/3}\gamma_{1}/2(1-x^{2})\times \]

\[ \times \int_{-1}^{1}\tilde{\varphi}(\xi)(1-\xi^{2})^{-1/3} (1+x^{2}-2x\xi)^{-7/6}\,d\xi . \]

Hence, according to (1.11), we have \(F(x,0)\in C^{\infty}(|x|<1)\), and \(F''(x,0)\) as \(x\to \pm1\) may tend to \(\infty\) of order \(\leq 2-2\chi_{1}\), if \(\chi_{1}<1\). Further, by virtue of (2.12′), it is clear that \(F(x,y)\) admits, in \(\Omega\cup\{y=0,\ |x|<1\}\), derivatives of arbitrary order with respect to \(x\). Relying on these properties of the function \(F\), one can readily verify that the function \(Z(x,0)\) from (2.13) has the following properties: 1) \(Z(x,0)\in C^{1}(|x|\leq 1)\) and at the points \(x=\pm1\) has a zero of order \(>1\); 2) \(Z(x,0)\in C^{\infty}(|x|<1)\); 3) \(Z''(x,0)\) as \(x\to\pm1\) may tend to \(\infty\) of order not higher than logarithmic.

1. Let there exist a solution of Problem 1. Then (2.26) holds, and the following two fundamental relations between \(\tau(x)\) and \(\nu(x)\), brought from the hyperbolic part \(\tilde{\Delta}\cup\tilde{\Delta}^{*}\) of the mixed domain \(\tilde{D}\) to the line of type change \(y=0\), hold:

\[ \tau(x)=\gamma\int_{0}^{x}\nu(t)(x-t)^{-1/3}P\bigl((x-t)^{4/3}\bigr)\,dt+\Psi_{1}(x), \qquad 0<x<x_{0}, \tag{2.27} \]

\[ \tau(x)=\gamma\int_{x}^{1}\nu(t)(t-x)^{-1/3}P\bigl((x-t)^{4/3}\bigr)\,dt+\Psi_{2}(x), \qquad x_{0}<x<1, \tag{2.28} \]

where 1) \(P\bigl((x-t)^{4/3}\bigr)\) admits derivatives of arbitrary order for \(x\ne t\), with \(P(0)=1\), and the first derivatives for \(x=t\) tend to zero of order not lower than \(1/3\); 2) \(\Psi_i\) is a function depending only on \(\psi_i,\ i=1,2;\ \Psi_1\in C^{6}(0<x\leq x_{0})\), \(\Psi_2\in C^{6}(x_{0}\leq x<1)\); 3) \(\Psi_1'\) (\(\Psi_2'\)) in the worst case behaves like \(x^{\chi_{2}-1}\) \(\bigl((1-x)^{\chi_{2}-1}\bigr)\), when \(x\to0\) \((x\to1)\); 4) \(2\Gamma(5/6)\Gamma(1/3)\gamma=(4/3)^{1/3}\Gamma(1/6)\).

The formulated result is contained in [2, 3].

§ 3. 1. Let us prove the existence of a solution of the system (2.26), (2.27), (2.28). For this purpose we eliminate \(\tau(x)\) from (2.26), (2.27). Then

\[ \int_{0}^{x}\nu(t)(x-t)^{-1/3}P\bigl((x-t)^{4/3}\bigr)\,dt =H_{1}(x)+ \]

\[ +\int_{0}^{1}\nu(t)[L(x,t)+l(x,t)]\,dt, \tag{3.1} \]

where

\[ \gamma H_{1}(x)=\Phi(x)-\Psi_{1}(x),\qquad l(x,t)=(t+x-2tx)^{-1/3}-|t-x|^{-1/3}. \]

Equation (3.1) (we regard the right-hand side as known for the time being) is a generalized Abel equation. By a known method (see [7]) it is reduced to a Volterra integral equation of the second kind

\[ \frac{2\pi}{\sqrt{3}}\,\nu(x)+\int_{0}^{x}\nu(t)(x-t)^{1/3}P_1\bigl((x-t)^{2/3}\bigr)\,dt = \]

\[ = \overline{H}_1(x)+\overline{L}(x)+\overline{l}(x), \tag{3.2} \]

where

\[ \overline{L}(x)=\int_{0}^{1}\nu(t)L_0(x,t)\,dt,\qquad L_0(x,t)= \]

\[ = \frac{d}{dx}\int_{0}^{x}\frac{L(\xi,t)\,d\xi}{(x-\xi)^{2/3}}, \tag{3.3} \]

\[ \overline{H}_1(x)=\frac{d}{dx}\int_{0}^{x}\frac{H_1(t)\,dt}{(x-t)^{2/3}}, \]

\[ \overline{l}(x)=\frac{d}{dx}\int_{0}^{x}\int_{0}^{1}\frac{\nu(t)l(\xi,t)}{(x-\xi)^{2/3}}\,dt\,d\xi, \]

and \(P_1\in C(0\le x,t\le 1)\), and has the same smoothness (in the sense of differentiability) as its argument \((x-t)^{2/3}\), with \(P_1(0)\ne 0\). It is known [8] that

\[ \overline{l}(x)=-\frac{\pi}{\sqrt{3}}\,\nu(x)-\int_{0}^{1}\nu(t)K^{-}(x,t)\,dt, \]

where

\[ K^{-}=(t/x)^{2/3}[1/(t-x)-1/(x+t-2tx)], \]

and the integral is understood in the sense of the Cauchy principal value. Consequently, (2.3) can be rewritten in the form

\[ \nu(x)-\lambda\int_{0}^{x}\nu(t)(x-t)^{1/3}P_1\bigl((x-t)^{2/3}\bigr)\,dt=\rho(x),\qquad \lambda\pi\sqrt{3}=-1, \tag{3.4} \]

\[ \rho(x)/\lambda=-\overline{H}_1(x)+\int_{0}^{1}\nu(t)K^{-}(x,t)\,dt-\int_{0}^{1}\nu(t)L_0(x,t)\,dt. \]

After inverting equation (3.4) and making simple transformations, it is easy to see that

\[ \nu(x)=-\lambda\int_{0}^{1}\nu(t)K^{-}(x,t)\,dt=h_1^*(x)+ \]

\[ +\int_{0}^{1}\nu(t)L_1^*(x,t)\,dt,\qquad 0<x<x_0, \]

where

\[ h_1^*(x)=-\lambda\overline{H}_1(x)-\lambda^2\int_{0}^{x}\overline{H}_1(\xi)(x-\xi)^{1/3}Q_1\bigl((x-\xi)^{2/3}\bigr)\,d\xi, \]

\[ L_1^*(x,t)=-\lambda L_0(x,t)-\lambda^2\int_{0}^{x}L_0(\xi,t)(x-\xi)^{1/3}\times \]

\[ \times Q_1\bigl((x-\xi)^{2/3}\bigr)\,d\xi+\lambda^2\int_{0}^{x}K^{-}(\xi,t)\times \]

\[ \times (x-\xi)^{1/3}Q_1\bigl((x-\xi)^{2/3}\bigr)\,d\xi; \]

\((x-t)^{1/3}Q_1\) is the resolvent of the kernel \((x-t)^{1/3}P_1\).

Similarly, from (2.26)—(2.28) we find

\[ \nu(x)+\lambda\int_0^1 \nu(t)K^+(x,t)\,dt = h_2^*(x)+\int_0^1 \nu(t)L_2^*(x,t)\,dt, \tag{3.5} \]

where

\[ K^+(x,t)=[(1-t)/(1-x)]^{2/3}\,[1/(t-x)+1/(t+x-2tx)],\qquad x_0<x<1, \]

and \(h_2^*, L_2^*\) are known functions depending on \(\Psi_2\) and \(L\), respectively.

Exclusively for convenience, we rewrite (3.4), (3.5) in the form

\[ S_\nu^i \equiv \nu_i(x)+(-1)^i\lambda\int_{-1}^1 \nu_i(t)K_i(x,t)\,dt = \]

\[ = h_i(x)+\int_{-1}^1 \nu_i(t)L_i(x,t)\,dt, \tag{3.6} \]

where

\[ \nu_1(x)=\nu_2(x)=\nu\bigl((x+1)/2\bigr),\quad x\ne x^0=2x_0-1,\quad h_i(x)= \]

\[ = h_i^*\bigl((x+1)/2\bigr),\qquad K_i(x,t)=[(1-(-1)^i t)/(1-(-1)^i x)]^{2/3}\times \]

\[ \times [\,1/(t-x)+(-1)^i/(1-tx)\,], \]

\[ L_i^*(x/2+1/2,\ t/2+1/2)=2L_i(x,t),\qquad i=1,2. \]

As is known [9], the singular integral equation (3.6) in the case when \(L_i\equiv 0\) can be reduced to an equivalent Fredholm integral equation of the second kind. The method used for this purpose will below be employed to regularize equation (3.6).

1. Let

\[ A_i(x)=h_i(x)+\int_{-1}^1 \nu_i(t)L_i(x,t)\,dt. \]

Define the function \(A^i(x)\) as follows: \(A^1=A_1,\ A^2=p_1\), if \(-1<x<x^0\); \(A^1=p_2,\ A^2=A_2\), if \(x^0<x<1\). Here \(p_i(x)\) is a certain function, Hölder-continuous in the domain of definition.

Equation (3.6) can be rewritten in the form \(S_\nu^i=A^i\). Invert the operator \(S^i\) in the class of the sought \(\nu\). Then

\[ 4\nu_i(x)=3A^i(x)-(-1)^i3\lambda\int_{-1}^1 A^i(t)K^i(x,t)\,dt, \tag{3.7} \]

where

\[ K^i=[(1-t^2)/(1-x^2)]^{1/3}\,[1/(t-x)+(-1)^i/(1-tx)]. \]

Since \(\nu_1(x)=\nu_2(x)\) for \(|x|<1,\ x\ne x^0\), from (3.7) we have

\[ p_2^1(x)=R_2^1(x)\pm\lambda\int_{-1}^1 p_2^1(t)K_1^2(x,t)\,dt, \tag{3.8} \]

where

\[ R_2^1(x)=A_2^1(x)\pm\lambda\int_{-1}^1 A_2^1(t)K_2^1(x,t)\,dt \]

and the upper sign is taken for \(-1\le x\le x^0\), the lower for \(x^0\le x\le 1\). Here and below \(M_j^i(x)=M_i(x)\), if \(-1<x<x^0\); \(M_j^i(x)=M_j(x)\), if \(x^0<\)

\(< x < 1;\quad K_j^i(x,t)=K_i,\) if \(-1<t<x^0;\quad K_j^i(x,t)=K_j,\) if \(x^0<t<1\). As Gellerstedt showed, the singular integral equation (3.8) is equivalent to the Fredholm equation

\[ p_2^1(x)+\lambda\int_{-1}^{1} p_2^1(s)\,\overline{K}_1^2(x,s)\,ds=\chi_2^1(x), \tag{3.9} \]

where

\[ \frac{4}{3}\chi_2^1=R_2^1 \pm \lambda\int_{-1}^{1}R_2^1(s)\bigl[(1-s^2)/(1-x^2)\bigr]^{1/3}\times \]

\[ \times \overline{\Gamma}_2^1(x,s)\,ds \equiv E(R_2^1), \tag{3.10} \]

\[ \overline{\Gamma}^{1}(x,s)=\bigl[(x^0-s)(1+x)/|x^0-x|(1+s)\bigr]^{1/3}\times \]

\[ \times [1/(s-x)+1/(1-sx)], \]

\[ -\overline{\Gamma}^{2}(x,s)=\bigl[(x^0-s)(1-x)/|x^0-x|(1-s)\bigr]^{1/3}\times \]

\[ \times [1/(s-x)-1/(1-sx)]; \]

\(\overline{K}_1^2\) is a regular kernel and admits a majorant of the form \(M|x^0-x|^{-1/3}\). This kernel can be written explicitly, and therefore it is not difficult to see that, for \(-1<x,s<1,\ x,s\ne x^0\), it has derivatives of all orders.

From the uniqueness of the solution of the Gellerstedt problem for the equation \(Tz=0\) in the domain \(\widetilde{D}\) with data on \(\widetilde{\sigma}_1\) and on the characteristics \(\widetilde{A}_4\widetilde{D}_0,\widetilde{A}_3\widetilde{B}\), there follows the unconditional solvability of equation (3.9). Let \(\Gamma_2^1\) be its resolvent. Then

\[ p_2^1(x)=\chi_2^1(x)-\lambda\int_{-1}^{1}\chi_2^1(s)\Gamma_2^1(x,s)\,ds\equiv \overline{E}(\chi_2^1). \tag{3.11} \]

Next one can write

\[ R_2^1(x)=H_2^1(x)+\int_{-1}^{1}v_1(s)l_2^1(x,s)\,ds, \]

where

\[ H_i(x)=h_i(x)-(-1)^i\lambda\int_{-1}^{1}h_2^1(t)K_2^1(x,t)\,dt= \]

\[ =\widetilde{E}(h_i),\qquad l_i(x,s)=\widetilde{E}(L_i). \]

Consequently, in view of (3.10),

\[ \frac{4}{3}\chi_2^1(x)=\overline{H}_2^1(x)+\int_{-1}^{1}v_1(t)\overline{l}_2^1(x,t)\,dt, \]

where \(\overline{H}_2^1(x)=E(H_2^1)\), \(\overline{l}_2^1(x,t)=E(l_2^1)\). Taking the latter into account in (3.11), as a result we obtain

\[ p_2^1(x)=\overline{h}_2^1(x)+\int_{-1}^{1}v_2(t)\overline{L}_2^1(x,t)\,dt, \tag{3.12} \]

where

\[ 4\overline{h}_2^1(x)=3\overline{E}(\overline{H}_2^1),\qquad 4\overline{L}_2^1(x,t)=3\overline{E}(\overline{l}_2^1). \]

Thus,

\[ A^1(x)=\widetilde{h}^1(x)+\int_{-1}^{1}v_1(t)L^1(x,t)\,dt, \tag{3.13} \]

where \(h^1(x)=h_1(x)\), \(L^1(x,t)=L_1(x,t)\), if \(-1<x<x^0\); \(h^1(x)=\overline{h}_2(x)\), \(L^1(x,t)=\overline{L}_2(x,t)\), if \(x^0<x<1\).

Substitute expression (3.13) into (3.7) \((i=1)\). As a result, after interchanging the order of integration in the repeated improper integral, where one integral is ordinary, we obtain

\[ \nu_1(x)-\int_{-1}^{1}\nu_1(t)\,\widetilde{L}(x,t)\,dt=\widetilde{h}(x), \tag{3.14} \]

where

\[ \frac{4}{3}\,\widetilde{L}(x,t)=L^1(x,t)+ \]

\[ +\lambda\int_{-1}^{1} L^1(s,t)K^1(x,s)\,ds \equiv U(L^1),\qquad \widetilde{h}=U(h^1). \]

The integral equation (3.14) (as to the properties of the kernel and of the free term, see below) is a Fredholm equation, and in the class of sought solutions it is equivalent to (3.6). Since the solution of problem 1 is unique, (3.14) is certainly solvable.

1. It is not difficult to verify that \(h_1(x)\in C^5(-1<x\le x^0)\), \(h_1(x)=O(1)(1+x)^{1/3}\), \(h_2(x)\in C^5(x^0\le x<1)\), \(h_2(x)=O(1)(1-x)^{1/3}\).

Let \(-1<x<x^0\). Then

\[ H_2^1(x)=h_1(x)+\lambda\int_{-1}^{x^0} h_1(t)\left[(1-t^2)/(1-x^2)\right]^{1/3}\times \]

\[ \times\left[1/(t-x)-1/(1-tx)\right]\,dt+ \]

\[ +\lambda\int_{x^0}^{1} h_2(t)\left[(1-t^2)/(1-x^2)\right]^{1/3}\left[1/(t-x)+1/(1-tx)\right]\,dt. \]

Hence, on the basis of the known properties of the Cauchy-type integral (see [10], §§ 21, 22), we conclude: 1) \(H_2^1(x)\in C^{(4,\delta)}(-1<x<x^0)\), i.e. it admits a 4th-order derivative satisfying the Hölder condition with exponent \(\delta\), \(0<\delta<1\); 2) \(H_2^1\), as \(x\to -1\) \((x\to x^0)\), tends to \(\infty\) of order \(\le 1/3\) (of logarithmic order).

It is clear that \(H_2^1(x)\in C^{(4,\delta)}(x^0<x<1)\) and, at the point \(x=1\), behaves in the same way as at \(x=-1\).

If \(-1<x<x^0\), then obviously

\[ \overline{H}_2^1(x)=H_2^1(x)+\lambda\left(|x^0-x|(1-x)\right)^{-1/3}\times \]

\[ \times\int_{-1}^{x^0} H_2^1(s)\left[(x^0-s)(1-s)\right]^{1/3} \left(\frac{1}{s-x}+\frac{1}{1-sx}\right)\,ds+ \]

\[ +\lambda\left(|x^0-x|(1+x)\right)^{-1/3}\int_{x^0}^{1}\overline{H}_2^1(s)\left[(s-x^0)(1+s)\right]^{1/3}\times \]

\[ \times\left(\frac{1}{s-x}-\frac{1}{1-sx}\right)\,ds. \]

Consequently, \(\overline{H}_2^{\,1}(x)\in C^{(4,\delta)}(-1<x<x^0)\), and as \(x\to -1\), \(x^0\) may tend to \(\infty\) of order \(\leq 1/3\). It has these properties also for \(x^0<x<1\).

Using the properties of the functions \(F_2^1\), \(\overline{h}_2^{\,1}\), \(h^1\), we finally see that \(\widetilde h(x)\in C^{(4,\delta)}(|x|<1,\ x\ne x^0)\), and as \(x\to \pm1\), \(x^0\) may tend to \(\infty\) of order \(\leq 1/3\).

4) From (3.3), after differentiation under the integral sign, we have

\[ L_0(x,t)=\int_0^x L_\xi(\xi,t)(x-\xi)^{-2/3}\,d\xi . \]

The result proved in § 2, item 2, permits us to conclude that \(L_x\) is representable in the form \(L_x=\mu\ln|x-t|+\mathcal L^*(x,t)\), where 1) \(\mu=\mu_1=\mathrm{const}\) for \(x\leq t\), \(\mu=\mu_2=\mathrm{const}\) for \(x\geq t\); 2) \(\mathcal L^*\) is a certain function admitting derivatives of arbitrary order when \(0<x,t<1\), and on the line \(x=t\) the first derivatives may have only integrable singularities (see (2.24), (2.25)). Moreover, \(\mathcal L^*\) has the property that the singularities of its derivatives, if any, can be separated out in explicit form.

Relying on these properties of the function \(\mathcal L^*\) and on the properties of the function \(P\), from (2.27) it is not difficult to verify that 1) \(L_i(x,t)\in C(-1\leq x,t\leq 1)\), \(L_i(x,t)\in C^\infty(-1<x,t<1,\ x\ne t)\), and at \(x=t\) the first derivatives tend, generally speaking, to \(\infty\) as \(|x-t|^{-2/3}\ln|x-t|\); 2) the singularities in the derivatives of \(L_i\) can be separated out in explicit form; 3) the kernel \(\widetilde L\) of the integral equation (3.14) belongs to \(C^\infty(-1<x,t<1,\ x\ne t,\ x\ne x_0)\), and at the points \(x=\pm1\), \(x_0\) it may have fixed singularities of order \(\leq 1/3\), while its first derivatives at \(x=t\) may tend to \(\infty\) as \(|x-t|^{-2/3}\ln|x-t|\).

5) Let \(\mu(x)\in C^{(0,\delta)}(a\leq x\leq b)\). Then (see [8], p. 90)

\[ \frac{d}{dx}\int_a^b \mu(t)\ln|t-x|\,dt = -\int_a^b \mu(t)/(t-x)\,dt . \tag{3.15} \]

If, however, \(\mu(x)\in C^1(a<x<b)\), and \(\mu'(x)\) at the points \(x=a\), \(x=b\) has only a singularity of order \(<1\), then

\[ \int_a^b \mu(t)/(t-x)\,dt = \mu(b)\ln(b-x)- \]

\[ -\mu(a)\ln(x-a)-\int_a^b \mu'(t)\ln|t-x|\,dt . \tag{3.16} \]

It follows directly from (3.15), (3.16) that if \(\mu(x)\in C^{(m,\delta)}(a\leq x\leq b)\), and \(\mu^{(m)}(x)\) as \(x\to a,b\) does not tend to \(\infty\) of order \(\geq1\), then the Cauchy-type integral with density \(\mu\) also belongs to this class.

Relying on these facts, one can show that the solution \(\nu_i(x)\) of equation (3.6) belongs to \(C^4(|x|<1,\ x\ne x^0)\), and at the points \(x=\pm1\), \(x^0\) may tend to \(\infty\) of order \(\leq 1/3\). Consequently, \(\nu(x)\in C^4(0<x<1,\ x\ne x_0)\), and at \(x=0,x_0,1\) it may have a singularity of order \(\leq 1/3\).

After \(\nu(x)\) has been found by solving the Holmgren problem in the domain \(\widetilde D^{+}\) and the singular Tricomi problem in the domains \(\widetilde\Delta\), \(\widetilde\Delta^*\), we construct a solution \(u(x,y)\) of problem 1 in the domain \(D\). It should be noted that \(u(x,y)\), as a function of the corresponding characteristic coordinates inside the characteristic arcs \(A_1D_0\) and \(AB\), has at least the same degree of smoothness as \(\nu(x)\) [1, 2].

The possibility of constructing a solution \(u(x,y)\) of problem 1 in the remaining part \(D^*\setminus D\) of the domain \(D^*\) was proved in [2].

References

1. Nakhushev A. M. DAN SSSR, 170, No. 1, 38–40, 1966.
2. Nakhushev A. M. Siberian Mathematical Journal, 8, No. 1, 39–68, 1967.
3. Nakhushev A. M. DAN SSSR, 166, No. 3, 536–540, 1966.
4. Gellerstedt S. Sur un problème aux limites pour une équation linéaire aux dérivées partielles du second ordre de type mixte. Thèse, Uppsala, 1935.
5. Frankl F. I. Izv. AN SSSR, Mathematical Series, 8, No. 5, 195–224, 1945.
6. Levi E. E. Uspekhi Matematicheskikh Nauk, vol. 8, 249–292, 1940.
7. Goursat É. Course of Mathematical Analysis, 3, 2. Moscow, 1934.
8. Bitsadze A. V. Equations of Mixed Type. Moscow, Publishing House of the Academy of Sciences of the USSR, 1959.
9. Gellerstedt S. Arkiv f. M. A. O. F., 26A, No. 3, 1937.
10. Muskhelishvili N. I. Singular Integral Equations. Moscow, Fizmatgiz, 1962.

Received by the editors
May 20, 1966

Institute of Mathematics, Siberian Branch of the Academy of Sciences of the USSR