Skip to content
BY 4.0 license Open Access Published by De Gruyter August 26, 2021

Qualitative analysis for the nonlinear fractional Hartree type system with nonlocal interaction

  • Jun Wang EMAIL logo

Abstract

In the present paperwe study the existence of nontrivial solutions of a class of static coupled nonlinear fractional Hartree type system. First, we use the direct moving plane methods to establish the maximum principle(Decay at infinity and Narrow region principle) and prove the symmetry and nonexistence of positive solution of this nonlocal system. Second, we make complete classification of positive solutions of the system in the critical case when some parameters are equal. Finally, we prove the existence of multiple nontrivial solutions in the critical case according to the different parameters ranges by using variational methods. To accomplish our results we establish the maximum principle for the fractional nonlocal system.

MSC 2010: 35J61; 35J20; 35Q55; 49J40

1 Introduction and main results

In the present paper we study the coupled nonlinear fractional Hartree system in the following form

(1.1) ( Δ ) α 2 u = μ 1 R N | u ( y ) | p | x y | N γ d y | u | p 2 u + β R N | v ( y ) | p | x y | N γ d y | u | p 2 u , x R N , ( Δ ) a 2 v = μ 2 R N | v ( y ) | p | X y | N γ d y | v | p 2 v + β R N | u ( y ) | p | x y | N γ d y | v | p 2 v , x R N , u , v > 0  and  u , v C l o c 1 , 1 R N F α R N ,

where N3,γ(0,N),1pN+γNα and

(1.2) F α = u : R N R R N | u ( x ) | 1 + | x | N + α < .

Recall that the operator (Δ)α2 is the fractional Laplacian in RN which is defined as a nonlocal pseudo differential operator

(1.3) ( Δ ) α 2 u ( x ) = C N , α P . V R N u ( x ) u ( y ) | x y | N + α d y := C N , α lim ε 0 | x y | ε u ( x ) u ( y ) | X y | N + α d y

for each uCloc1,1(N)Fα(N), where the constant CN,α=(N1cos(2πω1)|ω|N+αdω)1 and ω = (ω1, · · ·, ωN)(see [3, 6, 7, 9, 49]). Note that both the fractional Laplacians (Δ)α2 and the Hartree type nonlinearities are nonlocal in the system (1.1). Moreover,we can check that the system is closely related to the following integral equation

(1.4) u ( x ) = μ 1 R N R N , α | u ( y ) | p 2 u ( y ) ϕ u ( y ) | x y | N α d y + β R N R N , α | u ( y ) | p 2 u ( y ) ϕ v ( y ) | x y | N α d y , v ( x ) = μ 2 R N R N , α | v ( y ) | p 2 v ( y ) ϕ v ( y ) | x y | N α d y + β R N R N , α | v ( y ) | p 2 v ( y ) ϕ u ( y ) | x y | N α d y ,

where ϕw(y)=N|w(z)|p|yz|Nγdz and RN,α=Γ(Nα2)πN/22αΓ(α2).

One of the motivation to study the system (1.1) is motivated by recent studies on the nonlinear fractional Schrödinger equation with nonlocal interaction

(1.5) i ψ t = ( Δ ) α 2 ψ + V ψ χ C ( x ) | ψ | p | ψ | p 2 ψ , x R N .

If α = 2, N = 3 and p = 2, the system 1.5 is related to the condensate in the mean field regime. Indeed, concerning the movement of the identical and non-relativistic basic particles (such as bosons or electrons), under the influence of an external potential, the interaction between two particles is governed by the nonlinear nonlocal Hatree equation(see [17, 18, 23, 24]). Here the function ψ is a radially symmetric two-body potential function defined and * denotes the convolution in R3. The most typical external potential is the Coulomb function C(x) = |x|−1. On the other hand, the system 1.5 is also used in the description of the Bose-Einstein condensates, in which V is the trapping potential and the nonlocal interaction also describes the interaction between the bosons in the condensate [14, 44, 47]. When V = 0, 1.5 is also known as nonlinear Choquard equation [33, 37, 40], and the equation 1.5 with V = 0 also arises from the model of wave propagation in a media with a large response length [1, 26]. Many papers considered the general interaction case. That is,

(1.6) C ( x ) = A γ | x | N γ and A γ = Γ N γ 2 Γ γ 2 π N / 2 2 γ .

Then the stationary system of 1.5 reduces to

(1.7) ( Δ ) α 2 ψ + λ ψ = μ C ( x ) | ψ | p | ψ | p 2 ψ , x R N .

If α = 2, the paper [41] proved the existence and some properties of solution of 1.7. Recently, the paper [20] prove the existence of nodal solution of 1.7. For more general information one can refer to the papers [19, 20, 30, 31, 40, 42, 46] and references therein. For the fractional case 0 < α < 2, the paper [12] proved the regularity and classification of the solution of 1.7. Recently, by using the direct moving plane methods, the paper [11] make the classification of the positive solution of 1.7 when p = 2 and Nγ = 2α. The paper [28] make the classification of the positive solution of 1.7 for the general case.

The system (1.1) was studied in the sevral recent papers [51, 52, 53, 54, 56] for the case α = 2. This kind of systems are considered in the basic quantum chemistry model of small number of electrons interacting with static nucleii which can be approximated by Hartree or Hartree-Fock minimization problems (see [29, 35, 39]). In fact, the Euler-Lagrange equations corresponding to such Hartree problem are

(1.8) Δ u i + V ( x ) u i + j i R 3 u j 2 ( y ) | x y | d y u i + ε i u i = 0 , x R 3 , 1 i k ,

where k ∈ N, V(x) describes the attractive interaction between the electrons and the nucleii, the integral term shows the repulsive Coulomb interaction between the electrons, and −εi are the Lagrange multipliers. Following the discussion in [39], we usually consider the case that some components are set to be equal in 1.8. For example, when k = 2 and u1 = u2, then 1.8 is reduced to a scalar equation

(1.9) Δ u + V ( x ) u + R 3 u 2 ( y ) | x y | d y u + ε u = 0 , x R 3 .

The solutions of 1.9 were considered in, for example, [19, 38, 39]. We notice that in 1.8, the interaction between electrons is repulsive while the one in 1.5 is attractive. On the other hand, if k = 4, u1 = u2 and u3 = u4, then we can also obtain (1.1)when V = 0, p = q = 2, N = 3 and γ = α = 2. Recent years, the following nonlocal system

(1.10) ε 2 Δ u + λ 1 ( x ) u = μ 1 ϕ u | u | p 2 u + β ϕ v | u | p 2 u , x R N , ε 2 Δ v + λ 2 ( x ) v = μ 2 ϕ v | v | p 2 v + β ϕ u | v | p 2 v , x R N ,

has been studied by many literatures, where ϕu(x) = RN C(xy)|u(y)|p dy and C(x) is given in 1.6. Note that the system 1.10(or (1.1)) has two semitrivial solutions (u, 0) and (0, v), where u, v are solutions of 1.7. In order to make clear statement one gives the following definitions. We say (u, v) is a nontrivial solution of 1.10(or (1.1)) if u ≠ 0 and v ≠ 0. If u, v > 0, we say (u, v) is a positive solution of 1.10(or (1.1)). A solution is called a nontrivial ground state solution(or positive ground state solution) if its energy is minimal among all the nontrivial solutions(or all the positive solutions) of 1.10(or (1.1)). The paper [56] considered the semi-classical case. Under some conditions for the potential function λi(x), i = 1, 2, the existence of a ground state solution of 1.10 for ε > 0 small and β > 0 sufficiently large was proved. Later, the paper [53] studied the case λi(x) = λi = constant, ε = 1, p = 2, N = 3 and γ = 2. The authors proved the existence and nonexistence of positive ground state solutions of 1.10. Moreover, various qualitative properties of ground state solutions are also obtained. Recently, the papers [51, 54] studied the existence and properties of normolized solution of 1.10 with general interaction. Very recently, the paper [52] proved the existence of nontrivial solutions of the more general nonlocal interaction case of 1.10.

Motivated by the previous works, in the present paper we shall study the general case of the system 1.10 in fractional situation. Precisely, the main purpose of this paper is the following three parts. First, we use the direct moving plane methods to establish the maximum principle(Decay at infinity and Narrow region principle) and prove the symmetry and nonexistence of positive solution of this nonlocal system. Second, we make complete classification of positive solutions of the system for the critical case when μ1 = μ2. Finally, we prove the existence of multiple nontrivial solutions of the critical case for μ1μ2 by using variational methods. To accomplish the first two conclusions, we shall use a variant (for nonlocal nonlinearity) of the direct method of moving planes for fractional Laplacians due to the paper [6, 7] to obtain symmetry, monotonicity, nonexistence and classification of the positive solutions to (1.1). The methods of moving planes was initially invented by Alexanderoff in the early 1950s. Later, it was further developed by Serrin [48], Gidas, Ni and Nirenberg [21, 22], Caffarelli, Gidas and Spruck [2], Chen and Li [4], Li and Zhu [32] and many others. For more literatures on this direction, one can see the papers [4, 7, 8, 13, 16, 32, 36, 55] and the references therein. In this paper we establish the maximum principle(Decay at infinity and Narrow region principle, see Theorems 2.6-2.7 below) for the fractional nonlocal system. This is new and we believe that it will be useful to study the other nonlocal problems. To accomplish the third result, we should make careful study the uniqueness of the synchronous solutions of (1.1) and use the perturbation methods to obtain the nontrivial ground state solution of (1.1).

Then we first have the following main results for the subcritical and critical cases.

Theorem 1.1

Assume that N ≥ 2, 0 < α < 2, 0 < γ < N and μ1, μ2, β > 0. Then the system (1.1) has no positive solution for 1p<N+γNα. If p=N+γNαand μ1=μ2, then every positive solution (u, v) of (1.1) must has the following form u=v=(μ1+β)Nα2(γ+α)μNα2U(μ(xy)), where U(x)=τ0(11+|x|2)Nα2and I(k)=πN/2Γ(N2k2)Γ(Nk), where τ0=(RN,αI(Nα2)I(Nγ2))Nα2(γ+α).

From the results of Theorem 1.1, we have the following corollary.

Corollary 1.2

If u = v, μ1 = μ2 = 1 and p=N+γNα, then the following critical Hartree type equation

(1.11) ( Δ ) α 2 u = R N | u ( y ) | N + γ N α | X y | N γ d y | u | 2 α + γ N N α u , i n R N

has a unique positive solution of the form

(1.12) U μ ( x ) = C μ 1 + μ 2 x x 0 2 N α 2 f o r s o m e μ > 0 a n d x 0 R N .

Remark 1.3

The result of Corollary 1.2 has been obtained recently by [11, 28].

Next we consider the case p=N+γNα,μ1μ2 and β ∈ R. Note that the system (1.1) has two semitrivial solutions (u, 0) and (0, v), where

(1.13) u ( x ) = μ 1 N α 2 ( γ + α ) μ N α 2 U ( μ ( x y ) )  and  v ( x ) = μ 2 N α 2 ( γ + α ) μ N α 2 U ( μ ( x y ) ) ,

and U is given in Theorem 1.1. We define the space

(1.14) D 1 , α R N := u L 2 N N α R N : ( Δ ) α 4 u L 2 R N 2 <

with the norm |u|D1,α2=| (Δ)α4u |L2(N)2. As in the [15, Proposition 3.6], we know that

(1.15) | u | D 1 , α 2 = ( Δ ) α 4 u L 2 R N 2 = C ( N , α ) 2 R N R N | u ( x ) u ( y ) | 2 | x y | N + α d y d y ,

where C(N,α)=(N1cosζ1ζN+αdζ)1. We infer from Hardy-Littlewood-Sobolev inequality(see Lemma 2.1 below) and the embedding D1,α(N)L2NNα(N) that there exists S > 0 such that

(1.16) | u | D 1 , α 2 = | ( Δ ) α 4 u | L 2 R N 2 S R N R N | u ( x ) | p | u ( y ) | p | x y | N γ d x d y 1 p ,

where p=N+γNα. Since Uμ satisfies the equation 1.11, it follows that

(1.17) U μ D 1 , α 2 = ( Δ ) a 4 U μ L 2 R N 2 = R N R N U μ ( x ) p U μ ( y ) p | x y | N γ d x d y = S p p 1 = S N + γ γ + α .

Obviously, the system (1.1) has two semitrivial solutions (μ1Nα2(γ+α)Uμ,0) and (0,μ2Nα2(γ+α)Uμ). In order to find the nontrivial solutions of (1.1), we define Dα:=D1,α(N)×D1,α(N) and the functional

(1.18) J ( u , v ) = 1 2 | u | D 1 , α 2 + | v | D 1 , α 2 1 2 p R N μ 1 ϕ u | u | p + μ 2 ϕ v | v | p + 2 β ϕ v | u | p .

For β < 0, we consider the set

(1.19) N = ( u , v ) D α : u 0 , v 0 , ϑ ( u , v ) ( u , 0 ) = J ( u , v ) ( 0 , v ) = 0 .

It is clear that any nontrivial solutions of (1.1) belong to N and N ≠ (see below). We define

(1.20) K = inf ( u , v ) N J ( u , v ) = inf ( u , v ) N 1 2 1 2 p | u | D 1 , α 2 + | v | D 1 , α 2 = inf ( u , v ) N 1 2 1 2 p R N μ 1 ϕ u | u | p + μ 2 ϕ v | v | p + 2 β ϕ v | u | p .

Considering the following nonlinear problem

(1.21) μ 1 k 2 p 2 + β l p k p 2 = 1 , μ 2 l 2 p 2 + β k p l p 2 = 1 , k , l > 0.

Then we have the following existence results.

Theorem 1.4

Assume that p=N+γNα. Then the following conclusions hold true.

  1. If β < 0, then the infimum K can not be attained.

  2. If β > 0, then (1.1) has a nontrivial least energy solution (U, V) with J(U, V) = K.

  3. There exists 0 < β0 < (p − 1) max1, μ2} small such that for 0 < β < β0, there exists solution (k(β), l(β)) of 1.21. Moreover, (k(β), l(β)) satisfies

    J(U,V)<J(k(β)U1,l(β)U1),β(0,β0).

    This implies that (k(β)U1,l(β)U1) ) is a different positive solution of (1.1) from (U, V).

  4. If 2α + γ > N, then there exists β1 > β0 such that K = J(k0U1, l0U1) for β > β1. If 2α + γ < N and β ≥ (p − 1) max1, μ2}, then K = J(k1U1, l1U1), where (k0, l0) and (k1, l1) satisfy 1.21. If 2α + γ = N and β ∈ (0, min1, μ2}) (max1, μ2},∞), then K=J(μ2βμ1μ2β2U1,μ1βμ1μ2β2U1).

Remark 1.5

  1. The condition 2α + γN(or 2α + γ < N) equals to p ≥ 2(or p < 2) in Theorem 1.4 (iv).

  2. In Theorem 1.4 (iii), we obtain two different solutions to the system (1.1). That is, the nontrivial ground state solution and the positive solution of (k(β)U1,l(β)U1). Due to the fractional nonlocal operator, we do not know wether the solution (U, V) is positive and (U,V)=(k(β)U1,l(β)U1). We shall pursue this question in the future.

  3. Combining Theorems 1.1 and 1.4, we know that if μ1 = μ2, then k0=k2=l0=l2=(μ1+β)Nα2(γ+α), where k0, k2, l0, l2 are given in Theorem 1.4.

2 Main conclusions from the direct methods of moving planes

Throughout the paper, we use the following notations:

  1. | · |p is the norm of Lp(RN) defined by |u|p=(N|u|p)1/p for 0<p<.

  2. Let c > 0 be an arbitrary constants.

In this section we shall establish the main maximum principle theorems(Decay at infinity and Narrow region principle) for the fractional nonlocal system by using the direct methods of moving planes. We first recall the following classical Hardy-Littlewood-Sobolev inequality(see [34, Theorem 4.3]) and Cauchy-Schwarz inequality(see [34, Theorem 5.9] or [20, Inequality 3.3]) for nonlocal problem.

Lemma 2.1

  1. Assume that f ∈ Lp1 (RN) and g ∈ Lq1 (RN). Then one has

    R N R N f ( x ) g ( y ) | x y | t d x d y c p 1 , q 1 , t | f | p 1 | g | q 1 ,

    where 1 < p1, q 1 < ∞, 0 < t < N and 1p1+1q1+tN=2.

  2. If fLp1(N), then

    (N(Nf(y)|xy|Nγdy)q1)1q1c|f|p1,
  3. where 1q1=1p1γN.

    NN|u(x)|p|v(y)|p|xy|Nγdxdy(NN|u(x)|p|u(y)|p|xy|Nγdx)12(NN|v(x)|p|v(y)|p|xy|Nγdx)12,

    where γ ∈ (0, N) and p ≥ 1.

For 0 < α < 2, we define

(2.1) H α R N := u L 2 R N : | u ( x ) u ( y ) | | x y | N + α 2 L 2 R N ,

which is endowed with the natural norm

(2.2) u α := R N R N | u ( x ) u ( y ) | 2 | x y | N + α d x d y + R N | u | 2 d x 1 2 .

From [15, Theorem 6.5], we know that the embedding Hα(N)Lq(N)(q[ 2,2α ]) and 2α=2NNα(N3). From Hardy-Littlewood-Sobolev inequality Lemma 2.1 (i) and (iii), we know that if uHα(N), then the nonlinearity of the system (1.1) belongs to Lp(RN) for each 1pN+γNα. Hence if 1p<N+γNα, we call it a subcritical case in (1.1). If p=N+γNα, we say it a critical case in (1.1).

Next we shall study the properties of the positive solution (u, v) of (1.1). To describe the asymptotic behavior of u and v at infinity, we introduce the Kelvin transform ˜u, ˜v of u, v center at 0 defined by

(2.3) u ~ ( x ) = 1 | x | N α u x | x | 2  and  v ~ ( x ) = 1 | x | N α v x | x | 2

for each xN\{0}. It’s obvious that the Kelvin transform u˜,v˜ may have singularity at 0. Moreover we have lim|x||x|Nαu˜(x)=u(0)>0 and lim|x||x|Nαv˜(x)=u(0)>0. We infer from the definition 2.3 and u,vCloc1,1(N)Fα(N) that u˜,v˜Cloc1,1(N)Fα(N), and the integral property N|u(x)|N+γNα|x|Nγdx< equals to N|u˜|N+γNα(x)dx<. According to the system (1.1), we know that N|u(x)|N+γNα|x|Nγdx<. This conclusion holds similarly for ˜v. Now we are ready to calculate the system for (u˜,v˜). That is, one infers from 1.3 that

(2.4) (Δ)α2u~(x)=CN,αP.VRNux|x|21|x|Nα1|y|Nα+1|y|Nαux|x|2uy|y|2|xy|N+αdy=ux|x|2(Δ)α21|x|Nα+CN,αP.VRNux|x|2u(y)|y|Nα|xy/|y2N+αdy=1|x|N+α(Δ)α2ux|x|2=μ1|x|N+αRN|u(y)|p|x/|x2yNγdyux|x|2p2ux|x|2+β|x|N+αRN|v(y)|p|x/|x2yNγdyux|x|2p2ux|x|2=μ1|x|α+γ(p1)(Nα)RN|y|p(Nα)(N+γ)|u~(y)|p|Xy|Nγdy|u~(x)|p2u~(x)+β|x|α+γ(p1)(Nα)RN|y|p(Nα)(N+γ)|v~(y)|p|xy|Nγdy|u~(x)|p2u~(x).

Thus, we know that (u˜,v˜) is the positive solution of

(2.5) ( Δ ) α 2 u ~ = μ 1 | x | m R N | u ~ ( y ) | p | y | m | χ y | N γ d y u ~ ( x ) p 1 + β | x | m R N | v ~ ( y ) | p | y | m | χ y | N γ d y u ~ ( x ) p 1 , x R N { 0 } , ( Δ ) α 2 v ~ = μ 2 | x | m R N | v ~ ( y ) | p | y | m | X y | N γ d y v ~ ( x ) p 1 + β | x | m R N u ~ ( y ) p | y | m | x y | N γ d y v ~ ( x ) p 1 , x R N { 0 } ,

where m := α + γp(Nα). Thus, we see that if m > 0, then ( 1p<N+γNα subcritical case). If m = 0, then p=N+γNα (critical case).

Before going further, we need the following maximum principle for - α2 superharmonic functions and Liouville theorem for α2- harmonic functions. For the details of the proof one can refer to the papers [7, 49, 57].

Lemma 2.2

  1. (Maximum Principle)Let Ω ⊂ RN be a bounded domain. Assume that uCloc1,1(N)Fα(N) and is lower semicontinuous on If Ω¯.

    { (Δ)α2u(x)0,xΩ,u(x)0,xN\Ω,

    then u(x) ≥ 0 for all x ∈ RN. Furthermore, if u = 0 at some point in Ω, then u(x) = 0 almost everywhere in RN. In addition, these conclusions hold for unbounded region Ω if we further assume that

    lim|x|u(x)0
  2. (Liouville Theorem)Let uCloc1,1(N)Fα(N) be solution of

    { (Δ)α2u(x)0,xN,u(x)0,xN.

    Then u ≡ C ≥ 0.

Next we prove the equivalence between (1.1) and 1.4.

Lemma 2.3

Assume that N ≥ 3, 0 < α < 2, γ ∈ (0, N), p ≥ 1 and uCloc1,1(N)Fα(N). If u is nonnegative solution of (1.1), then u satisfies the integral system 1.4, and vice versa.

Proof. Let G R α be the Green’s function for (Δ)α2 on BR(0) which is defined by

GRα(x,y)=CN,α|xy|Nα0tRsRlα21(1+l)N2dl for x,yBR(0) and GRα(x,y)=0 if x or yN\BR(0),

where sR=|xy|2R2 and tR=(1|x|2R2)(1|y|2R2) (see [27]). For each R > 0, we define

(2.6) w R ( x ) = B R ( 0 ) G R α ( x , y ) μ 1 R N R N , α u ( y ) p 1 ϕ u ( y ) | x y | N α d y + β R N R N , α u ( y ) p 1 ϕ v ( y ) | x y | N α d y d y , z R ( x ) = B R ( 0 ) G R α ( x , y ) μ 2 R N R N , α v ( y ) p 1 ϕ v ( y ) | x y | N α d y + β R N R N , α v ( y ) p 1 ϕ u ( y ) | x y | N α d y .

By using the properties of Green’s function, we can derive that (wR(x), zR(x)) satisfies

(2.7) Δ w R ( x ) = μ 1 R N R N , a u ( y ) p 1 ϕ u ( y ) | x y | N α d y + β R N R N , α u ( y ) p 1 ϕ V ( y ) | x y | N α d y , x B R ( 0 ) Δ z R ( x ) = μ 2 R N R N , α v ( y ) 1 ϕ V ( y ) | x y | N α d y + β R N R N , α v ( y ) 1 ϕ u ( y ) | x y | N α , x B R ( 0 ) , w R ( x ) = z R ( x ) = 0 , x R N B R ( 0 ) .

Let w˜R=uwR and z˜R=vzR. We infer from (1.1) and 2.7 that

(2.8) Δ w ~ R ( x ) = Δ z ~ R ( x ) = 0 , x B R ( 0 ) , w R ( x ) , z R ( x ) 0 , x R N B R ( 0 ) .

One deduces from the Maximum principle(Lemma 2.2) that foe any R > 0

(2.9) w ~ ( x ) = u ( x ) w R ( x ) 0  and  z ~ ( x ) = v ( x ) z R ( x ) 0  for all  x R N .

For each fixed x ∈ RN, we have that

(2.10) u ( x ) w ( x ) := μ 1 R N R N , α u ( y ) p 1 ϕ u ( y ) | x y | N α d y + β R N R N , α u ( y ) p 1 ϕ v ( y ) | x y | N α d y , v ( x ) z ( x ) := μ 2 R N R N , α v ( y ) p 1 ϕ v ( y ) | x y | N α d y + β R N R N , α v ( y ) p 1 ϕ u ( y ) | x y | N α ,

as R → ∞. On the other hand, it is easy to see that (w, z) is a solution of

(2.11) Δ w ( x ) = μ 1 u ( y ) p 1 ϕ u ( y ) + β u ( y ) p 1 ϕ v ( y ) , x R N , Δ z ( x ) = μ 2 v ( y ) p 1 ϕ v ( y ) + β v ( y ) p 1 ϕ u ( y ) , x R N .

Set ˜w = uw and ˜z = vz. Hence we have

(2.12) Δ w ~ ( x ) = Δ z ~ ( x ) = 0 , x R N , w ( x ) , z ( x ) 0 , x R N .

We infer from Lemma 2.2 (ii) that ˜w = uw = C0 ≥ 0 and ˜z = vz = C1 ≥ 0. This implies that

(2.13) u ( x ) = μ 1 R N R N , α u ( y ) p 1 ϕ u ( y ) | x y | N α d y + β R N R N , α u ( y ) p 1 ϕ v ( y ) | x y | N α d y + C 0 , v ( x ) = μ 2 R N R N , α v ( y ) p 1 ϕ v ( y ) | x y | N α d y + β R N R N , α v ( y ) p 1 ϕ u ( y ) | x y | N α + C 1 ,

Since μi , β ≥ 0 and μi + β > 0(i = 1, 2), we know that

(2.14) > u ( 0 ) w ( 0 ) = μ 1 R N R N , α C 0 2 p 1 | y | N α | y z | N α d y d z + β R N R N , α C 0 p 1 C 1 p | y | N α | y z | N α d y d z , > v ( 0 ) z ( 0 ) = μ 2 R N R N , α C 1 2 p 1 | y | N α | y z | N α d y d z + β R N R N , α C 1 p 1 C 0 p | y | N α | y z | N α d y d z .

This implies that C0 = C1 = 0. Thus, we arrive at 1.4. Conversely, we follow the idea of [5, 8, 57] to infer that if (u, v) is a nonnegative solution of 1.4, then (u, v) is also a solution of (1.1).

The next lemma state the basic properties for the Kelvin transform u˜,v˜Cloc1,1(N)Fα(N).

Lemma 2.4

The Kelvin transformu, ˜v) satisfies the integral system

(2.15) u ~ ( x ) = μ 1 R N R N , α | x y | N α R N | u ~ ( z ) | p | z | m | y z | N γ d z u ~ ( y ) p 1 | y | m d y + β R N R N , α | x y | N α R N | v ~ ( y ) | p | z | m | y z | N γ d z u ~ ( y ) p 1 | y | m d y , x R N { 0 } , v ~ ( x ) = μ 2 R N R N , α | x y | α R N | n ~ ~ ( z ) | p | z | m | y z | N γ d z n ~ ( y ) p 1 | y | m d y + β R N R N , α | x y | N α R N | u ~ ( y ) | p | z | m | y z | N γ d z v ~ ( y ) p 1 | y | m d y , x R N { 0 } ,

where m = N + γp(Nα). Furthermore, we have

(2.16) lim inf | x | 0 u ~ ( x ) > 0 a n d lim inf | x | 0 v ~ ( x ) > 0.

Proof. A direct computation shows that

(2.17) μ 1 R N R N , α | x y | N α R N | u ~ ( z ) | p | z | m | y z | N γ d z u ~ ( y ) p 1 | y | m d y + β R N R N , α | x y | N α R N | v ~ ( z ) | p | z | m | y z | N γ d z u ~ ( y ) p 1 | y | m d y = μ 1 R N R N , α x y | y | 2 N α R N | z | m | z | N γ u ( z ) p y | y | 2 z | z | 2 N γ d z | z | 2 N | y | m | y | N γ u ( y ) p 1 d y | y | 2 N + β R N R N , α x y | y | 2 N γ R N | z | m | z | N γ v ( z ) p y | y | 2 z | z | 2 N γ d z | z | 2 N | y | N γ u ( y ) p 1 d y | y | 2 N = μ 1 | x | N γ R N R N , α x | x | 2 y N α R N u ( z ) p y z N γ d z u ( y ) p 1 d y + β | x | N γ R N R N , α x | x | 2 y N γ R N v ( z ) p | y z | N γ d z u ( y ) p 1 d y = 1 | x | N γ u x | x | 2 = u ~ ( x ) .

Similarly, we can prove the second equality in 2.15. Next we shall prove 2.16. As in [11, Ineqaulity 2.19](or [28]), one can prove that Fu,FvL(B2(0)). Let χC0(N) satisfy 0 ≤ χ ≤ 1, χ = 1 in ∞B1(0) and χ = 0 in RN \ B2(0). We define

(2.18) ψ ( x ) = μ 1 R N , α R N χ ( y ) u ( y ) p 1 ϕ u ( y ) | x y | N α d y + β R N , α R N χ ( y ) u ( y ) p 1 ϕ v ( y ) | x y | N α d y .

Then we know that ψ(x) satisfies(see [5, 8, 57])

(2.19) ( Δ ) α 2 ψ ( x ) = μ 1 χ ( y ) u ( y ) p 1 ϕ u ( y ) + β χ ( y ) u ( y ) p 1 ϕ v ( y ) , x R N .

We infer from the properties of χ, μ1 + β > 0 and 2.18 that

(2.20) R N , α | x | N α μ 1 ϕ u u p 1 L 1 B 1 ( 0 ) + β ϕ v u p 1 L 1 B 1 ( 0 ) ψ ( x ) C N , α 2 | x | N α ,

where CN,α2>0 is a positive constant depending on N and α. This implies that

(2.21) C N , α 1 | x | N α ψ ( x ) C N , α 2 | x | N α ,

where CN.α1>0 is a positive constant. For each R > 0, we let h R ( x ) = u ( x ) ψ ( x ) + C N , α 2 R N α . Thus, it follows from (1.1) and 2.19 that

(2.22) ( Δ ) α 2 h R ( x ) = ( 1 χ ( x ) ) μ 1 u ( y ) p 1 ϕ u ( y ) + β u ( y ) p 1 ϕ v ( y ) 0 , x B R ( 0 ) , h R ( x ) C N , α 2 R N α ψ ( x ) , x R N B R ( 0 ) .

We infer from Lemma 2.2 that hR ≥ 0 in RN for each R > 0 large. Particularly, we have

(2.23) u ( x ) ψ ( x ) + C N , α 2 R N α 0  in  B R ( 0 ) .

For arbitrarily fixed x, letting R → ∞in 2.23, we get uψ in RN. Hence we get that

(2.24) u(x)CN,α1|x|Nα

for |x| large enough. Thus, we deduce from 2.24 that

(2.25) lim inf | x | 0 u ~ ( x ) = lim inf | x | 0 1 | x | N α u x | x | 2 = lim inf | x | | x | N α u ( x ) C N , α 1 > 0.

Similarly, we can prove lim inf|x|→0 ˜v(x) > 0. This finishes the proof.

Now we are ready to use the method of moving planes. For each λ ∈ RN, we define

(2.26) T λ = x R N : x 1 = λ  and  Σ λ = x R N : x 1 < λ .

Let

(2.27) xλ:=(2λx1,x)=(2λx1,x2,,xN)

be the reflection of x about the plane Tλ, and define wλ(x)=u˜(xλ)u˜(x) and zλ(x)=v˜(xλ)v˜(x). We first need to prove that for λ sufficiently negative

(2.28) w λ ( x ) , z λ ( x ) 0 , x Σ λ 0 λ .

Then we can start moving plane Tλ from near x1 = −∞ to the right as long as 2.28 holds, until its limiting position and finally derive the symmetry. To accomplish this we set

(2.29) Σ λ , w = x Σ λ { 0 } : w λ ( x ) < 0  and  Σ λ , z = x Σ λ { 0 } : z λ ( x ) < 0 .

It suffices to show that Σλ,w=Σλ,z=0. To this purpose we calculate the term (Δ)α2wλ and (Δ)α2zλ on Σλ,w and Σλ,z respectively. Let

(2.30) P u ~ ( x ) = 1 | x | N γ u ~ p | x | m , P v ~ ( x ) = 1 | x | N γ v ~ p | x | m , G u ( x ) = Σ λ , w u ~ ( y ) p 1 w λ ( y ) | x y | N γ | y | m d y  and  G V ( x ) = Σ λ , z v ~ ( y ) p 1 z λ ( y ) | x y | N γ | y | m d y .

Then we have the following conclusion.

Lemma 2.5

Assume that λ ≤ 0. Then we have the following conclusions.

  1. If wλ(x) < 0 and zλ(x) ≥ 0, then we have

    (2.31) ( Δ ) α 2 w λ ( x ) max { p 1 , 1 } μ 1 P u ~ ( x ) + β P v ~ ( x ) u ~ ( x ) p 2 | x | m w λ ( x ) + μ 1 p G u ( x ) u ~ ( x ) p 1 | x | m , ( Δ ) α 2 z λ ( x ) β p G u ( x ) v ~ ( x ) p 1 | x | m .
  2. If wλ(x) ≥ 0 and zλ(x) < 0, then we have

    (2.32) ( Δ ) α 2 w λ ( x ) β p G V ( x ) u ~ ( x ) p 1 | x | m , ( Δ ) α 2 z λ ( x ) max { p 1 , 1 } μ 2 P v ~ ( x ) + β P u ~ ( x ) v ~ ( x ) p 2 | x | m z λ ( x ) + μ 2 p G V ( x ) V ~ ( x ) p 1 | x | m .
  3. If wλ(x) < 0 and zλ(x) < 0, then we have

    (2.33) ( Δ ) α 2 w λ ( x ) max { p 1 , 1 } μ 1 P u ~ ( x ) + β P v ~ ( x ) u ~ ( x ) p 2 | x | m w λ ( x ) + μ 1 p G u ( x ) u ~ ( x ) p 1 | x | m + β p G v ( x ) u ~ ( x ) p 1 | x | m , ( Δ ) α 2 z λ ( x ) max { p 1 , 1 } μ 2 P v ~ ( x ) + β P u ~ ( x ) v ~ ( x ) p 2 | x | m z λ ( x ) + μ 2 p G v ( x ) v ~ ( x ) p 1 | x | m + β p G u ( x ) v ~ ( x ) p 1 | x | m .

Proof. We only give the proof of the case (3), other cases can be proved similarly. We infer from 2.5 that

(2.34) ( Δ ) α 2 w λ = ( Δ ) α 2 u ~ ( x λ ) ( Δ ) α 2 u ~ ( x ) = μ 1 P u ~ ( x λ ) u ~ ( x λ ) p 1 | x λ | m + β P v ~ ( x λ ) u ~ ( x λ ) p 1 | x λ | m μ 1 P u ~ ( x ) u ~ ( x ) p 1 | x | m β P v ~ ( x ) u ~ ( x ) p 1 | x | m μ 1 P u ~ ( x λ ) u ~ ( x λ ) p 1 | x | m + β P v ~ ( x λ ) u ~ ( x λ ) p 1 | x | m μ 1 P u ~ ( x ) u ~ ( x ) p 1 | x | m β P v ~ ( x ) u ~ ( x ) p 1 | x | m = μ 1 P u ~ ( x ) u ~ ( x λ ) p 1 u ~ ( x ) p 1 | x | m + μ 1 P u ~ ( x λ ) P u ~ ( x ) u ~ ( x λ ) p 1 | x | m + β P v ~ ( x ) u ~ ( x λ ) p 1 u ~ ( x ) p 1 | x | m + β P v ~ ( x λ ) P v ~ ( x ) u ~ ( x λ ) p 1 | x | m = μ 1 L 1 + μ 1 L 2 + β L 3 + β L 4 .

We first give the estimate the L1. Since λ ≤ 0 and wλ , zλ ≤ 0, it follows that

(2.35) L 1 μ 1 P u ~ ( x ) max { p 1 , 1 } u ~ ( x ) p 2 u ~ x λ u ~ ( x ) | x | m = μ 1 max { p 1 , 1 } P u ~ ( x ) u ~ ( x ) p 2 | x | m ,

where we used the basic inequality

(2.36) a q b q max { q , 1 } b q 1 ( a b )  for  q 0  and  0 < a b .

Similarly we estimate the term L2 as follows.

(2.37) L 2 = R N u ~ ( y ) p | x λ y | N γ | y | m d y R N u ~ ( y ) p | x y | N γ | y | m d y u ~ ( x λ ) p 1 | x | m = Σ λ 1 | x y | N γ 1 | x λ y | N γ u ~ ( y λ ) p | y λ | m u ~ ( y ) p | y | m d y u ~ ( x λ ) p 1 | x | m Σ λ 1 | x y | N γ 1 | x λ y | N γ u ~ ( y λ ) p u ~ ( y ) p | y | m d y u ~ ( x λ ) p 1 | x | m Σ λ 1 | x y | N γ u ~ ( y λ ) p u ~ ( y ) p | y | m d y u ~ ( x λ ) p 1 | x | m Σ λ 1 | x y | N γ p u ~ ( y ) p 1 u ~ ( y λ ) u ~ ( y ) | y | m d y u ~ ( x λ ) p 1 | x | m = p G u ( x ) u ~ ( x ) p 1 | x | m ,

where we used the following basic inequality

(2.38) a q b q q b q 1 ( a b )  for  q 1  and  0 < a < b .

We can similarly get the estimates of L3, L4. Thus, we get the first inequality of 2.32. By using similar arguments one can obtain the second inequality of 2.32.

Then we have the following decay at infinity maximum principle for the nonlocal problem (1.1).

Theorem 2.6

Assumeλ < 0 such that wλFαCloc1,1λ,wor zλFαCloc1,1(Σλ,z) and the negative minimum of wλ or zλ is attained in the interior of Σλ \ {0}. Then, there exists some R0 > 0(depending on u, v, but is independent ofλ)such that, if x0Σλ,wor x0Σλ,z satisfying wλ(x0)=minΣλ,wwλ(x)<0 or zλ(x0)= minΣλ,zzλ(x)<0, then |x0|R0.

Proof. We divide into the following three cases to prove the conclusions.

Case 1. wλ(x0) < 0 and zλ(x0)≥ 0 for X0Σλ,w. Then we know that (wλ , zλ) satisfies 2.31. We infer from the definition of 1.3 and wλ(yλ) = −wλ(y) that

(2.39) ( Δ ) α 2 w λ ( x 0 ) = C N , α P . V . R N w λ ( x 0 ) w λ ( y ) | x 0 y | N + α d y = C N , α P . V . Σ λ w λ ( x 0 ) w λ ( y ) | x 0 y | N + α d y + R N Σ λ w λ ( x 0 ) w λ ( y ) | x 0 y | N + α d y C N , α Σ λ w λ ( x 0 ) w λ ( y ) | x 0 y λ | N + α d y + Σ λ w λ ( x 0 ) w λ ( y λ ) | x 0 y λ | N + α d y = 2 C N , α w λ ( x 0 ) Σ λ 1 | x 0 y λ | N + α d y .

For each fixed λ, we know that B| x0 |(x˜)N\Σλ for x˜=(x01+4| x0 |,(x0)), where x0=(x01,(x0)). Hence it follows that

(2.40) Σ λ 1 x 0 y λ N + α d y = R N Σ λ 1 x 0 y N + α d y B x 0 x ~ 1 x 0 y N + α d y B x 0 x ~ 1 5 N + α x 0 N + α d y = ω N 5 N + α x 0 α ,

where ωN denotes the volume of the unit ball in RN. Combining 2.39 and 2.40, we know that

(2.41) ( Δ ) α 2 w λ x 0 ω N 5 N + α x 0 α w λ x 0 .

On the other hand, it follows from wλ(x0)=minΣλ,wwλ(x) that

(2.42) (Δ)α2wλ(x0)A(x0)wλ(x0),

where

(2.43) A ( x ) = max { p 1 , 1 } μ 1 P u ~ ( x ) + β P v ~ ( x ) u ~ ( x ) p 2 | x | m + μ 1 p H u ( x ) u ~ ( x ) p 1 | x | m

and

(2.44) H u ( x ) = R N u ~ ( y ) p 1 | x y | N γ | y | m d y .

We claim that for x ∈ Σλ

(2.45) A(x)C|x|α+σ,

where σ=min{ α,N(11p)+mp } and C > 0 is independent of x and λ. Indeed, it is clear that

(2.46) u ~ ( x ) , v ~ ( x ) c | x | N α  for  x R N B R ( 0 )  and  R > 0.

Moreover, we infer from [28, Proposition 9] that

(2.47) P u ~ ( x ) , P v ~ ( x ) c | x | N γ  and  H u ( x ) c | x | N γ N m p .

This implies that

(2.48) A ( x ) c | x | N γ + m + ( p 2 ) ( N α ) + c | x | N γ N m p + m + ( p 1 ) ( N α ) c | x | 2 α + c | x | α + N 1 1 p + m p .

Hence we get the claim 2.45. Combining 2.41-2.42 and 2.45, we know that

(2.49) 0 ω N 5 N + α χ 0 α C χ 0 α + σ w λ x 0 .

If |x0| → ∞, we know that the right hand of 2.49 is strictly less than zero. This is a contradiction. Thus there exists R0 > 0 such that |x0|R0.

Case 2. wλ(x0)≥ 0 and zλ(x0)< 0 for x0Σλ,z We infer from 2.32 that

(2.50) ( Δ ) α 2 z λ ( x 0 ) A 1 ( x 0 ) z λ ( x 0 ) ,

where

(2.51) A 1 ( x ) = max { p 1 , 1 } μ 2 P v ~ ( x ) + β P u ~ ( x ) v ~ ( x ) p 2 | x | m + μ 2 p H v ( x ) v ~ ( x ) p 1 | x | m

and

(2.52) H u ( x ) = R N v ~ ( y ) p 1 | x y | N γ | y | m d y .

Similar to Case 1, we know that

(2.53) 0 ω N 5 N + α x 0 α C x 0 α + σ z λ x 0 .

Hence we get |x0|R0 for some R0 > 0.

Case 3. wλ(x0)< 0 and zλ(x0)< 0 for x0Σλ,wor x0Σλ,z. One deduces from 2.33 that

(2.54) ( Δ ) α 2 w λ x 0 A 3 x 0 w λ x 0  and  ( Δ ) α 2 z λ x 0 A 4 x 0 z λ x 0 ,

where

(2.55) A 3 ( x ) = max { p 1 , 1 } μ 1 P u ~ ( x ) + β P v ~ ( x ) u ~ ( x ) p 2 | x | m + μ 1 p H u ( x ) u ~ ( x ) p 1 | x | m + β p H v ( x ) u ~ ( x ) p 1 | x | m , A 4 ( x ) = max { p 1 , 1 } μ 1 P v ~ ( x ) + β P u ~ ( x ) v ~ ( x ) p 2 | x | m + μ 2 p H v ( x ) v ~ ( x ) p 1 | x | m + β p H u ( x ) v ~ ( x ) p 1 | x | m .

One deduces from 2.41 and 2.54 that

(2.56) ω N 5 N + α X 0 α A 3 ( x ) w λ x 0 + ω N 5 N + α X 0 α A 4 ( x ) z λ x 0 0.

Since wλ(x0), zλ(x0)< 0, we infer that at least one of

(2.57) ω N 5 N + α x 0 α A 3 ( x ) 0  or  ω N 5 N + α x 0 α A 4 ( x ) 0

holds. However, similar to the estimates in 2.45, we know that

(2.58) A3(x),A4(x)C|x|α+σ.

Thus we infer that

(2.59) ω N 5 N + α x 0 α A 3 ( x ) > 0  and  ω N 5 N + α x 0 α A 4 ( x ) > 0 ,

as |x0| → ∞. This is a contradiction. Hence we get |x0|R0 for some R0 > 0.

Next we prove the narrow region principle for the system (1.1).

Theorem 2.7

Let Ω{ xN:λl<x1<l } be a narrow region in Σλ \ {0λ}, where λ < 0 and l > 0 small.

Assume that (wλ,zλ)FαCloc1,1 satisfies the following conditions:

  1. Lemma 2.5 holds with domain in ΩΣλ;

  2. the negative minimum of wλ or zλ is attained in the interior of Σλ\{ 0λ } if Σλ0;

  3. the negative minimum of wλ or zλ can not be attained in (Σλ\{ 0λ })\Ω.

Then there exists l0 > 0(depending on λ continuously) sufficiently small such that for all 0 < ll0,

(2.60) w λ ( x ) 0 a n d z λ ( x ) 0 , x Ω .

Proof. We use the contradiction argument. If 2.60 does not holds, then there exists an x0Ω¯ such that

(2.61) w λ x 0 = min x Σ λ { 0 } w λ ( x ) < 0  or  z λ x 0 = min x Σ λ { 0 } z λ ( x ) < 0.

We divide into the following three cases to prove our conclusion.

Case 1. wλ(x0)=minxΣλ\{0}wλ(x)<0 and zλ(x0)≥ 0. Similar to 2.39, we know that x0(ΩΣλ) { xN:λl<x1<λ } such that

(2.62) ( Δ ) α 2 w λ x 0 2 C N , α w λ x 0 Σ λ 1 x 0 y λ N + α d y .

Let Al={ yN:l<y1x01<1,| y(x0) |<1 }Σλ. By using the similar arguments as in [7, Inequality (22)], one sees that

(2.63) Σ λ 1 x 0 y λ N + α d y A l 1 x 0 y λ N + α d y c l 1 1 s 1 + α d s c l α ,

as l → 0. On the other hand, as in 2.42-2.45 we know that

(2.64) (Δ)α2wλ(x0)A(x0)wλ(x0),

where

A ( x ) = max { p 1 , 1 } μ 1 P u ~ ( x ) + β P v ~ ( x ) u ~ ( x ) p 2 | x | m + μ 1 p H u ( x ) u ~ ( x ) p 1 | x | m

and

Hu(x)=Nu˜(y)p1|xy|Nγ|y|mdy.

Moreover, we have

(2.65) A ( x ) C | x | α + σ ,  where  σ = min α , N 1 1 p + m p .

Since Ω ⊂ {x ∈ RN : |x|λ}, it follows from 2.65 that for each x ∈ Ω

(2.66) A(x)C|λ|α+σ:=Dλ.

We infer from 2.62-2.63 that there exists l0 > 0 such that for 0 < ll0

(2.67) 2 e N , α Σ λ 1 x 0 y λ N + α d y 2 D λ .

Combining 2.64-2.66, we obtain that

(2.68) 0 w λ x 0 2 C N , α Σ λ 1 x 0 y λ N + α d y A x 0 w λ x 0 D λ < 0.

This is a contradiction.

Case 2. zλ(x0)=minxΣλ\{0}zλ(x)<0 and wλ(x0)≥ 0. As in 2.50-2.52 we infer that

(2.69) ( Δ ) α 2 z λ ( x ) A 1 x 0 z λ x 0 ,

where A1 and H u are given in 2.51-2.52. Moreover, we have

(2.70) A 1 ( x ) C | x | α + σ ,  where  σ = min α , N 1 1 p + m p .

By using the same arguments as in Case 1, one can find the contradiction.

Case 3. wλ(x0)=minxΣλ\{0}wλ(x)<0 and zλ(x0)<0 or zλ(x0)=minxΣλ\{0}zλ(x)<0 and wλ(x0) < 0. As in 2.54 we know that

(2.71) ( Δ ) α 2 w λ x 0 A 3 x 0 w λ x 0  and  ( Δ ) α 2 z λ x 0 A 4 x 0 z λ x 0 ,

where A3, A4 are given in 2.55. Furthermore, one has

(2.72) A 3 ( x ) , A 4 ( x ) C | x | α + σ ,  where  σ = min α , N 1 1 p + m p .

Thus, we get

(2.73) 2 C N , α Σ λ 1 x 0 y λ N + α d y A 3 x 0 w λ x 0 + 2 C N , α Σ λ 1 x 0 y λ N + α d y A 4 x 0 z λ x 0 0.

Since wλ(x0), zλ(x0) < 0, we know that at least one of

(2.74) 2 C N , α Σ λ 1 x 0 y λ N + α d y A 3 x 0 0  or  2 C N , α Σ λ 1 x 0 y λ N + α d y A 4 x 0 0

holds. However, as in the cases 1-2, one can infer that

(2.75) 2 C N , α Σ λ 1 x 0 y λ N + α d y A 3 x 0 > 0  and  2 C N , α Σ λ 1 x 0 y λ N + α d y A 4 x 0 > 0.

This is a contradiction.

3 Proof of Theorem 1.1

In this section we shall use the method of moving plane to prove the main results of Theorem 1.1. Precisely, we shall show the symmetry of u and v about Tλ0(λ0=0) by moving plane Tλ along x1 direction from −∞ to the right. We first divide into the following three steps to prove the subcritical case 1p<N+γNα.

Step 1. We claim that for λ < 0 sufficiently negative,

(3.1) w λ ( x ) 0  and  z λ ( x ) 0 for  x Σ λ { 0 } .

We use the contradiction argument. Assume that there exists x0 ∈ Σλ \ {0} such that wλ(x0) < 0or zλ(x0) < 0. From (2.26), we deduce that there exist σ0 > 0 and ε > 0 small such that u˜(x)σ0 and v˜(x)σ0 for all x ∈ Bε(0) \ {0}. This implies that ˜u(x) ≥ σ0 and ˜v(x) ≥ σ0 for all x ∈ Bε(0λ) \ {0λ}. On the other hand, we infer from the definition of ˜u, ˜v that there exists Λ0 > 0 large such that u˜(x),v˜(x)<σ02 for λ < −Λ0. Hence we get

(3.2) w λ ( x ) = u ~ ( x λ ) u ~ ( x ) σ 0 2 and z λ ( x ) = v ~ ( x λ ) v ~ ( x ) σ 0 2 x B ε ( 0 λ ) { 0 λ } .

Since lim|x|wλ(x)=lim|x|zλ(x)=0, we infer from Theorem 2.6 that if x0Σλ such that wλ(x0)= minxΣλwλ(x)<0 or zλ(x0)=minxΣλzλ(x)<0, then |x0|R0. Thus, for λ<max{ R0,Λ0 } we get the contradiction. This implies (3.1).

Step 2. Step 1 provides a starting point, from which we can move the plane Tλ to the right as long as (3.1) holds to its limiting position. Let

(3.3) λ 0 = sup { λ 0 : w μ ( x ) 0  and  z μ ( x ) 0  for all  x Σ μ , μ λ } .

In the following we shall prove that

(3.4) λ 0 = 0 and w λ 0 ( x ) = z λ 0 ( x ) 0  for all  x Σ λ 0 .

We use the contradiction argument. Suppose that λ0 < 0. We show that the plane Tλ can be moved further right. To be more rigorous, there exists some ε > 0, such that for any λ(λ0,λ0+ε), we have

(3.5) w λ ( x ) 0 and z λ ( x ) 0  for all  x Σ λ .

This is a contradiction with the definition of λ0. Hence we must have λ0 = 0. We shall prove (3.5) by using the narrow region principle. To accomplish this we first claim that

(3.6) w λ 0 ( x ) > 0 and z λ 0 ( x ) > 0  for all  x Σ λ 0 { 0 λ } .

Since the proof is similar, we only prove wλ0(x)>0 for xΣλ0\{ 0λ }. We infer from Lemma 2.4 that there exists a constant σ0 > 0 such that

(3.7) w λ 0 ( x ) σ 0 > 0 x B | λ 0 | ( 0 λ 0 ) .

We infer from the integral equations (2.15) that

(3.8) w λ 0 ( x ) = μ 1 R N , α R N P u ~ ( y ) u ~ ( y ) p 1 | y | m | x λ 0 y | N α d y R N P u ~ ( y ) u ~ ( y ) p 1 | y | m | x y | N α d y + β R N , α R N P v ~ ( y ) u ~ ( y ) p 1 | y | m | x λ 0 y | N α d y R N P v ~ ( y ) u ~ ( y ) p 1 | y | m | x y | N α d y = μ 1 R N , α Σ λ 0 1 | x y | N α 1 | x λ 0 y | N α × P u ~ ( y λ 0 ) u ~ ( y λ 0 ) p 1 | y λ 0 | m P u ~ ( y ) u ~ ( y ) p 1 | y | m d y + β R N , α Σ λ 0 1 | x y | N α 1 | x λ 0 y | N α × P v ~ ( y λ 0 ) u ~ ( y λ 0 ) p 1 | y λ 0 | m P v ~ ( y ) u ~ ( y ) p 1 | y | m d y := μ 1 R N , α L 1 + β R N , α L 2

Next we show that L1, L2 > 0. We infer from Lemma 2.4 that for yB| λ0 |/4(0λ0)

(3.9) P u ~ ( y ) = R N u ~ p ( z ) | y z | N γ | z | m d z min B | λ 0 | ( 0 λ 0 ) u ~ p B | λ 0 | / 4 ( y ) 1 | y z | N γ | z | m d z min B | λ 0 | ( 0 λ 0 ) u ~ p 1 | λ 0 | m B | λ 0 | / 4 ( 0 ) 1 | z | N γ d z = c > 0.

Similar we obtain

(3.10) Pv˜(y)c>0.

Now we are ready to deduce the estimates for L1. We infer from (3.9) that

(3.11) L 1 = Σ λ 0 1 | x y | N α 1 | x λ 0 y | N α u ~ ( y λ 0 ) p 1 | y λ 0 | m u ~ ( y ) p 1 | y | m P u ~ ( y ) d y + Σ λ 0 1 | x y | N α 1 | x λ 0 y | N α u ~ ( y λ 0 ) p 1 | y λ 0 | m P u ~ ( y λ 0 ) P u ~ ( y ) d y Σ λ 0 1 | x y | N α 1 | x λ 0 y | N α 1 | y λ 0 | m 1 | y | m u ~ ( y ) p 1 P u ~ ( y ) d y + Σ λ 0 1 | x y | N α 1 | x λ 0 y | N α u ~ ( y λ 0 ) p 1 | y λ 0 | m × Σ λ 0 1 | y z | N γ 1 | y λ 0 z | N γ u ~ ( z λ 0 ) p 1 | z λ 0 | m u ~ ( z ) p 1 | z | m d z d y Σ λ 0 1 | x y | N α 1 | x λ 0 y | N α 1 | y λ 0 | m 1 | y | m u ~ ( y ) p 1 P u ~ ( y ) d y c B | λ 0 | / 2 ( 0 λ 0 ) 1 | x y | N α 1 | x λ 0 y | N α 1 | y λ 0 | m 1 | y | m d y > 0.

Similarly, one can prove L2 > 0. Hence we prove the claim (3.6). Furthermore, as in the proof of (3.11) we know that

(3.12) w λ 0 ( x ) c > 0  and  z λ 0 ( x ) c > 0  for  B ϱ ( 0 λ ) { 0 λ 0 } ,

where ϱ ∈ (0, 0|/2). Thus, we get

(3.13) u ~ ( x ) u ~ λ 0 ( x ) c > 0  and  v ~ ( x ) v ~ λ 0 ( x ) c > 0  for  B ϱ ( 0 ) { 0 } .

Since u˜,v˜ are uniformly continuous on Bϱ+| λ0 |/2(0λ0), one infers from (3.13) that there exists ε0 (0, 0|/4) sufficiently small such that for each λ[ λ0,λ0+ε0 )

(3.14) u ~ ( x ) u ~ λ 0 ( x ) c 2 > 0  and  v ~ ( x ) v ~ λ 0 ( x ) c 2 > 0  for  B ϱ ( 0 ) { 0 } .

This implies that for each λ ∈ [λ0, λ0 + ε0)

(3.15) w λ ( x ) c 2 > 0  and  z λ ( x ) c 2 > 0  for  B ϱ ( 0 λ ) { 0 λ 0 } .

Now we are ready to use the contradiction argument to give the proof of (3.5). Let λ ∈ [λ0, λ0 + ε1) where ε1 > 0 is sufficiently small parameter. Assume that there exists x0 ∈ Σλ \ {0} such that

(3.16) w λ ( x 0 ) = min Σ λ { 0 } w λ < 0 or z λ ( x 0 ) = min Σ λ { 0 } z λ < 0.

From Theorem 2.6 we infer that

(3.17) x0BR0(0).

On the other hand, we infer from (3.12) that there exists ς ∈ (0, c/2) such that

(3.18) w λ 0 ( x ) ς > 0  and  z λ 0 ( x ) ς > 0  for  B R 0 ( 0 ) Σ λ 0 l 0 / 2 ¯ B ϱ / 2 ( 0 λ 0 ) ,

By continuity, we may find ε1 (0, min0, l0/2, 0|/2, ϱ/4}) such that for each λ ∈ [λ0, λ0 + ε1)

(3.19) w λ ( x ) ς 2 > 0  and  z λ ( x ) ς 2 > 0  for  B R 0 ( 0 ) Σ λ 0 l 0 / 2 ¯ B ϱ / 2 ( 0 λ 0 ) .

Since 0λBϱ/2(0λ0) and Bϱ/2(0λ0)Bϱ(0λ), we infer from (3.15) and (3.19) that

(3.20) w λ ( x ) ς 2 > 0  and  z λ ( x ) ς 2 > 0  for  B R 0 ( 0 ) Σ λ 0 l 0 / 2 ¯ { 0 λ 0 } .

We infer from (3.17) and (3.20) that the negative minimum of wλ or zλ can not be attained in Σλ0l0/2¯\{ 0λ }. We take Ω=Σλ\Σλ0l0/2¯ in Theorem 2.7. Then it follows that

(3.21) w λ ( x ) 0  and  z λ ( x ) 0 , x Σ λ Σ λ 0 l 0 / 2 ¯ { 0 λ } .

Thus, we infer from (3.21) and the definition of λ0 that for λ ∈ [λ0, λ0 + ε1)

(3.22) w λ ( x ) 0  and  z λ ( x ) 0 , x Σ λ { 0 λ } .

This is contradicts with the definition of λ0. Hence we obtain

(3.23) λ 0 = 0 , w λ 0 ( x ) 0  and  z λ 0 ( x ) 0 , x Σ λ 0 .

Similarly, we can move the plane Tλ from +∞ to the left and show that

(3.24) λ 0 = 0  and  w λ 0 ( x ) = z λ 0 ( x ) 0  for all  x Σ λ 0 .

Step 3. We complete the proof. For any two points XiN(i=1,2), one can choose X 0 to be the midpoint, i.e., X0=X1+X22. Similar to Steps 1-2(replacing 0 by X0), one can prove that (˜u, ˜v) is radially symmetric about X0, and so (u, v). That is, we obtain (u(X1),v(X1))=(u(X2),v(X2)). This implies that (u, v) is constant. Since a positive constant does not satisfy (1.1). This proves the nonexistence of positive solutions for (1.1) when 1p<N+γNα. This finishes the proof for the subcritical case.

Finally, we make the classification of positive solutions in the critical case p=N+γNα and μ1=μ2. The next lemma finds the exact formula solution of (1.1).

Lemma 3.1

For each μ > 0 and y ∈ RN, we know that u=v=(μ1+β)Nα2(γ+α)μNα2U(μ(xy)) is a solution of (1.1), where U(x)=τ0(11+|x|2)Nα2 and I(k)=πN/2Γ(N2k2)Γ(Nk), where τ0=(RN,αI(Nα2)I(Nγ2))Nα2(γ+α).

Proof. From the invariance of the system (1.1), we may assume μ = 1 and y = 0. Moreover, by using the Fourier transforms of the kernels of Riesz and Bessel potentials, we infer from Lemm [12, Lemma 4.2] that

(3.25) R N 1 | x y | 2 s 1 1 + | y | 2 N s d y = I ( s ) 1 1 + | y | 2 s .

A direct computation shows that

(3.26) R N U p ( y ) | x y | N γ d y = R N τ 0 p | x y | N γ 1 1 + | y | 2 N + γ 2 d y = τ 0 N + γ N α I N γ 2 1 1 + | x | 2 N γ 2 .

Hence it follows that

(3.27) R N , α ( μ 1 + β ) 2 p 1 2 ( p 1 ) R N U p 1 ( y ) | x y | N α R N U p ( z ) | z y | N γ d z d y = R N , α τ 0 N + α + 2 γ N α ( μ 1 + β ) 2 p 1 2 ( p 1 ) I N γ 2 R N 1 | x y | N α 1 1 + | y | 2 N α 2 d y = R N , α τ 0 N + α + 2 γ N α ( μ 1 + β ) 2 p 1 2 ( p 1 ) I N γ 2 I N α 2 1 1 + | x | 2 N α 2 = τ 0 ( μ 1 + β ) 2 p 1 2 ( p 1 ) 1 1 + | x | 2 N α 2 = 1 ( μ 1 + β ) 2 p 1 2 ( p 1 ) U ( x ) .

Thus, we know that u=v=(μ1+β)12(p1)μNα2U(μ(xy)) is a solution of (1.1).

Lemma 3.2

Assume that N2,0<α<2,0<γ<N,μ1=μ2,β>0andp=N+γNα. Then every positive solution of (1.1) must assume the form

(3.28) u = v = ( μ 1 + β ) N α 2 ( γ + α ) μ N α 2 U ( μ ( x y ) ) f o r s o m e μ > 0 a n d y R N ,

where U is given in Lemma 3.1.

Proof. As in the subcritical case,we know that every positive solution of (1.1) is radially symmetric and monotone decreasing about some point x0 N. We claim that (u, v) has the desired asymptotic behavior at ∞. That is, it satisfies

(3.29) lim | x | | x | N α u ( x ) = u  and  lim | x | | x | N α v ( x ) = v ,

where u and v are some positive constants. We use the contradiction argument. Assume that (3.29) does not hold. Suppose that x1 and x2 be any two different points in ℝN and let x0 be the midpoint of the line segment x1x2¯. Consider the Kelvin type transform centered at x0

(3.30) z ( x ) = 1 | x x 0 | N α u x x 0 | x x 0 | 2 + x 0  and  w ( x ) = 1 | x x 0 | N α v x x 0 | x x 0 | 2 + x 0 .

Then z(x) and w(x) must have a singularity at x0. By using the same arguments as in the proof of the subcritical case, we can deduce that z, w must be radially symmetric and monotone decreasing about its singular point x0 and hence u(x1) = u(x2). Since x1 and x2 are arbitrarily chosen in ℝN, u must be constant, thus u ≡ 0, which contradicts with u > 0. Hence the claim (3.29) holds.

Now we borrow an idea of [5, 8, 43] to give the proof of this lemma. Assume that (u, v) is a nonnegative solution of (1.1). Then if x0 = 0, we get that

(3.31) u ( s 1 x + a ) = 1 | x | N α u s 1 x | x | 2 + a  and  v ( s 2 x + a ) = 1 | x | N α u s 2 x | x | 2 + a

for x ∈N \ {0} and a ∈ R, where s 1 = u ( x 0 ) / u 1 N α  and  s 2 = v ( x 0 ) / v 1 N α . Without loss of generality we assume that a = 0 and μ1 + β = 1. Since the proof is similar, we only give the proof for the first conclusion of (3.31). Let x0N\{0} be any fixed point and e=x0| x0 |. We define the function

(3.32) w ( x ) = 1 | x | N α u s x | x | 2 e .

Then we know that w(0)=sαNu=u(0)=w(e) and u=v=sNα2 w is also a positive solution of (1.1). Thus w is radially symmetric with respect to some point x˜ that lies on the hyperplane e+12e through 12e which is

perpendicular to e. Moreover, since u, v are radially symmetric about 0, it follows that for any 12<r<1 and x1,x2Br(0)Br(e), we can deduce from (3.32) that

(3.33) w ( x 1 ) = 1 | x 1 | N α u s x 1 | x 1 | 2 e = 1 | x 2 | N α u s x 2 | x 2 | 2 e = w ( x 2 ) .

Hence we obtain that w(x)=w(| x12e |) e+12e, on and x˜=e2 and w is actually radially symmetric about e2. It is clear that there exists σ(12,12) such that | X0 |=12+σ12σ. We infer from (3.32) that

(3.34) 1 | 1 2 σ | N α u s 1 2 + σ 1 2 σ e = w ( 1 2 σ ) e = w ( 1 2 + σ ) e = 1 | 1 2 + σ | N α u s 1 2 σ 1 2 + σ e .

This implies that

(3.35) u ( s x 0 ) = 1 | x 0 | N α u s x 0 | x 0 | 2 .

For the case a ≠0, we can consider u(· + a) instead of u itself. As in [8, Lemma 3.2], one can prove that

(3.36) u ( m ) u = v ( m ) v = R N , α I N α 2 I N γ 2 N α γ + α ,

where (u, v) is a positive solution to (1.1) with symmetric center m. Finally, we define

(3.37) u ^ ( x ) = χ 1 N α 2 u ( χ 1 1 x + x 0 )  and  v ^ ( x ) = χ 2 N α 2 v ( χ 2 1 x + x 0 ) ,

where χ1=(u(x0)/u)1Nα and χ2=(v(x0)/v)1Nα. By using the similar arguments as in [8, Section 3.1], we know that

(3.38) u ^ ( x ) = U ( x ) = v ^ ( x ) .

This finishes the proof.

4 Proof of Theorem 1.4

In this section we focus on the existence of solution of (1.1) when β ∈ R and μ1 ≠ μ2. In the following we use 1.11-1.12 to construct the synchronous solutions of (1.1). That is, the solution of the form

(4.1) (u,v)=(kU1,lU1).

Then 4.1 solves (1.1) if and only if (k, l) satisfies 1.21. In the following we first find the solution of 1.21. To this purpose we borrow an idea of [10] to define

(4.2) g 1 ( k , l ) = μ 1 k 2 p 2 + β l p k p 2 1 , k > 0 , l 0 , g 2 ( k , l ) = μ 2 l 2 p 2 + β k p l p 2 1 , l > 0 , k 0 , h 1 ( k ) = β 1 p k 2 p μ 1 k p 1 p , 0 < k μ 1 1 2 p 2 and h 2 ( l ) = β 1 p l 2 p μ 2 l p 1 p , 0 < l μ 2 1 2 p 2 ,

where p=N+γNα. By this definition we know that g1(k,h1(k))=g2(h2(l),l)=0. In the next lemma we find the solution (k, l) for 1.21.

Lemma 4.1

Assume that β. Then we have the following results.

  1. If 2α + γ = N, then we have (μ2βμ1μ2β2,μ1βμ1μ2β2) is a solution of 1.21 for β(μ1μ2,min{ μ1,μ2 }) (max{ μ1,μ2 },).

  2. If 2α + γ > N and β > 0, then 1.21 has a positive solution (k0, l0), which satisfies g2(k, h1(k)) > 0 for 0 < k < k0. Moreover, 1.21 has another positive solution (k1, l1), which satisfies g1(h2(l), l) > 0 for 0 < l < l0.

  3. If 2α + γ < N and β > 0, then 1.21 has a positive solution (k2, l2), which satisfies g2(k, h2(k)) < 0 for 0 < k < k2. Moreover, 1.21 has another positive solution (k3, l3), which satisfies g1(h2(l), l) < 0 for 0 < l < l3.

Proof. We first prove the conclusion (1). If 2α + γ = N, then we know that 1.21 equals

(4.3) μ 1 k 2 + β l 2 = 1 , μ 2 l 2 + β k 2 = 1 , k , l > 0.

Thus,  if β(μ1μ2,min{ μ1,μ2 })(max{ μ1,μ2 },), then (μ2βμ1μ2β2,μ1βμ1μ2β2) is a solution of 4.3.Next we prove the conclusion (2) and (3). It is clear that g1(k, l) = 0, k, l > 0 equals

(4.4) l = h 1 ( k ) = β 1 p k 2 p μ 1 k p 1 p , 0 < k μ 1 1 2 p 2 .

Substituting this into g2(k, l) = 0, then we obtain that

(4.5) μ 2 1 μ 1 k 2 p 2 β + β k 2 p 2 = 1 μ 1 k 2 p 2 β k 2 p 2 2 p p

In order to finish the proof we define

(4.6) F ( k ) = μ 2 β + β 2 μ 1 μ 2 β k 2 p 2 1 β k 2 p 2 μ 1 β 2 p p , 0 < k < μ 1 1 2 p 2 .

We first consider the case 2α + γ > N. This implies that p > 2 in this case. A direct computation shows that

(4.7) lim k 0 + F ( k ) = μ 2 β and lim k μ 1 1 2 p 2 F ( k ) =

This implies that there exists k0(0,μ112p2) such that F (k0) = 0 and F (k) > 0 for each k ∈ (0, k0). Hence g2(k, h1(k)) > 0 for 0 < k < k0. Set l0 = h1(k0). Then (k0, l0) is a solution of g1(k, l) = 0 and g2(k, l) = 0. Similarly, one can find the (k1, l1) satisfies the conclusion (2). Finally, if 2α + γ < N and β > 0, by using the similar argument as in the case (2), one can find (k2, l2) and (k3, l3), which satisfy the conclusion (3).

Next we study the uniqueness of (k0, l0) and (k2, l2).

Lemma 4.2

Assume that 2α + γ > N. Then (see Lemma 4.1 (2)) is the unique solution of 1.21 for 0<βμ1μ2. √k0, l0)(Furthermore, if (k, l) satisfies

(4.8) k 2 + l 2 k 0 2 + l 0 2 , g 1 ( k , l ) 0 , g 2 ( k , l ) 0 , k , l 0 , ( k 2 , l 2 ) ( 0 , 0 ) ,

then 4.8 has unique solution (k, l) = (k0, l0). On the other hand, if 2α + γ < N and β ≥ (p − 1) max1, μ2} then the equation 4.8(with (k0, l0) is replaced by (k2, l2)) has a unique solution (k, l) = (k2, l2).

Proof. We first consider the case 2α + γ > N, that is, p > 2. By 4.6, a direct computation shows that

(4.9) F ( k ) = ( 2 p 2 ) β 2 μ 1 μ 2 β k 2 p 3 ( 2 p ) ( 2 2 p ) p β k 1 2 p 1 β k 2 p 2 μ 1 β 2 p 1 ,

where 0<k<μ112p2. Hence we obtain that F(k)<0 for 0<k<μ112p2. Combining with 4.7 we obtain that F (k) = 0 has unique solution k0. This implies that (k0, l0) is the unique solution of 1.21. On the other hand, 0<kμ112p2 and 0<lμ212p2 a direct computation shows that for

g1(0,l)<0,g1(k,0)0,g2(0,l)0,g2(k,0)<0,g1l(k,l),g1k(k,l),g2l(k,l),g2k(k,l)>0.

This implies that g(k, l) and g2(k, l) are increasing in k, l respectively. Assume that k < k0. we infer from the monotone property of gi(k, l)(i = 1, 2, ) that

0g1(k,l0)<g1(k0,l0)=0.

This is a contradiction. Thus, we get kk0. Similarly, we can prove ll0. Hence we obtain (k, l) = (k0, l0). The rest of the conclusion follows from [10, Lemmas 2.2-2.4].

Remark 4.3. The condition 0<βμ1μ2 only is required for the uniqueness of (k0, l0). If (k, l) satisfies 4.8, we knew that (k, l) has unique solution (k0, l0) for each β > 0.

The next lemma states the relations between K and (1.1).

Lemma 4.4

If K is attained by a couple (u, v) N, then (u, v) is a critical point of J for −∞ < β < 0.

Proof. We infer from (u, v) N that

(4.10) | u | D 1 , α 2 = μ 1 R N ϕ u | u | p | β | R N ϕ u | v | p and | u | D 1 , α 2 = μ 1 R N ϕ u | u | p | β | R N ϕ u | v | p ,

where ϕu(x)=Nu(y)p|xy|Nγdy. If K = J(u, v), then there exists two Lagrangian multipliers λ1, λ2 R such that

(4.11) J ( u , v ) + λ 1 H 1 ( u , v ) + λ 2 H 2 ( u , v ) = 0 ,

where H1=J(u,v)(u,0) and H2=J(u,v)(0,v). Using (u, 0) and (0, v) as a test function, we obtain that

(4.12) λ 1 ( 2 p 2 ) μ 1 R N ϕ u | u | p + ( 2 p ) | β | R N ϕ u | v | p λ 2 | β | p R N ϕ u | v | p , λ 2 ( 2 p 2 ) μ 2 R N ϕ v | v | p + ( 2 p ) | β | R N ϕ u | v | p λ 1 | β | p R N ϕ u | v | p .

We infer from β < 0 and 4.10 that

(4.13) ( 2 p 2 ) μ 1 R N ϕ u | u | p + ( 2 p ) | β | R N ϕ u | v | p × ( 2 p 2 ) μ 2 R N ϕ v | v | p + ( 2 p ) | β | R N ϕ u | v | p > | β | p R N ϕ u | v | p 2 .

It follows that λ1 = λ2 = 0.

Now we are ready to give the proof of Theorem 1.4. We first consider the case β < 0.

Proof of Theorem 1.4 (i). Let wμi:=μiαN2(γ+α)U1(x)(i=1,2). It is clear that wμi (x) satisfies the equation

(4.14) ( Δ ) α 2 u = μ i R N | u ( y ) | N + γ N α | x y | N γ d y | u | 2 α + γ N N α u , in R N , i = 1 , 2.

We define e1=(1,,1)N and

(4.15) u R ( x ) , v R ( x ) = w μ 1 ( x ) , w μ 2 ( x + 4 R e 1 ) .

We first claim that

(4.16) lim R R N R N | u R ( x ) | N + γ N α | v R ( y ) | N + γ N α | x y | N γ d x d y = 0 , as R .

In fact, it is clear that

(4.17) R N × R N = B R ( 0 ) × B R ( 4 R e 1 ) B R ( 0 ) × R N B R ( 4 R e 1 ) ( ( R N B R ( 0 ) ) × ( B R ( 4 R e 1 ) ) ) R N B R ( 0 ) × R N B R ( 4 R e 1 ) := Ω 1 Ω 2 Ω 3 Ω 4 ,

where BR(x0) := {x ∈N : |xx0|R}. We shall prove that 4.17 holds in each domain Ωi(i = 1, 2, 3, 4). Since |xy| ≥ 2R for (x,y)BR(0)×BR(4Re1), it follows that

(4.18) B R ( 0 ) B R ( 4 R e 1 ) | u R ( x ) | N + γ N α | v R ( y ) | N + γ N α | x y | N γ d x d y 1 ( 2 R ) N γ B R ( 0 ) w μ 1 ( x ) N + γ N α d x B R ( 0 ) w μ 2 ( x ) N + γ N α d x c ( 2 R ) N γ .

On the other hand, we infer from Hardy-Littlewood-Sobolev inequality(see Lemma 2.1) that

(4.19) B R ( 0 ) R N B R ( 4 R e 1 ) | u R ( x ) | N + γ N α | v R ( y ) | N + γ N α | x y | N γ d x d y c B R ( 0 ) w μ 1 ( x ) 2 N N α d x N + γ 2 N R N B R ( 0 ) w μ 2 ( x ) 2 N N α d x N + γ 2 N 0 as R .

Similarly, one can prove that

(4.20) Ω 3 Ω 3 | u R ( x ) | N + γ N α | v R ( y ) | N + γ N α | x y | N γ d x d y 0 as R .

Combining 4.18-4.20, we obtain that the claim 4.16 holds.

Next we prove that for each β < 0, there exists positive (tR, sR) such that (tRuR, sRvR) N. Let

(4.21) A 1 = μ 1 R N R N | u R ( x ) | N + γ N α | u R ( y ) | N + γ N α | x y | N γ d x d y , A 2 = μ 2 R N R N | u R ( x ) | N + γ N α | u R ( y ) | N + γ N α | x y | N γ d x d y  and  A 3 = β R N R N | u R ( x ) | N + γ N α | v R ( y ) | N + γ N α | x y | N γ d x d y .

Then we infer from 4.16 that

(4.22) A 1 A 2 > A 3 2  for each  β < 0  and  R > 0  sufficiently large. 

It is clear that (tRuR, sRvR) N equals

(4.23) t R 2 p A 1 = t R p A 1 + s R p A 3  and  s R 2 p A 2 = s R p A 2 + t R p A 3 , t R , s R > 0.

Since p > 1, we infer from sRp=(tR2ptRp)A1/A3>0 and A3<0 that t > 1. Substituting this into 4.23, we obtain that

(4.24) G ( t R ) := A 2 A 1 | A 3 | ( 1 t R 2 2 p ) 2 p p A 1 A 2 | A 3 | 2 | A 3 | t R 2 p 2 + A 1 A 2 | A 3 | , t > 1.

We infer from p > 1 that G(1) = |A3| > 0 and limtRG(tR)=. Hence we know that there exists tR, sR > 0 such that (tRuR, sRvR) N. Furthermore, we claim that

(4.25) limR(| tR1 |+| sR1 |)=0.

To accomplish this we first prove that tR, sR are bounded. Assume that tR →∞as R → ∞.We infer from 4.23 that

(4.26) t R 2 A 1 = t R 2 p A 1 s R p t R p | A 3 | , s R 2 A 2 = s R 2 p A 2 s R p t R p | A 3 |

and

(4.27) t R 2 p A 1 t R 2 A 1 = s R 2 p A 2 s R 2 A 2 .

This implies that sR →∞as R → ∞. We deduce from p > 1 that

(4.28) t R p A 1 t R 2 2 p A 1 1 4 t R p A 1  and  s R p A 2 s R 2 2 p A 2 1 4 t R p A 2 for R large .

Then we obtain

(4.29) | A 3 | = t R p t R 2 p s R p A 1 t R p 4 s R p A 1  and  | A 3 | = s R p s R 2 p t R p A 2 s R p 4 t R p A 2 .

Combining 4.28-4.29 we get

(4.30) 0 < 1 16 A 1 A 2 | A 3 | 2 0 , as R .

This is a contradiction. Therefore sR, tR are bounded. We infer from 4.26 that 4.25 holds. Hence we infer from 1.17 that

(4.31) K J ( t R u R , s R v R ) = γ + α 2 ( N + γ ) t R 2 | u R | D 1 , α 2 + s R 2 | v R | D 1 , α 2 = γ + α 2 ( N + γ ) t R 2 μ 1 α N γ + α + s R 2 μ 2 α N γ + α S N + γ γ + α .

Thus, as R → ∞ in 4.31, we get that Kγ+α2(N+γ)(μ1αNγ+α+μ2αNγ+α)SN+γγ+α. On the other hand, we infer from β < 0 and 1.16 that

(4.32) | u | D 1 , α 2 μ 1 R N R N | u ( x ) | p | u ( y ) | p | x y | N γ d x d y μ 1 S p | u | D 1 , α 2 p .

This implies that |u|D1,α2μ1αNγ+αSN+γγ+α. Similarly, we can deduce that |v|D1,α2μ1αNγ+αSN+γγ+α. Hence we infer from 4.23 that

(4.33) K γ + α 2 ( N + γ ) μ 1 α N γ + α + μ 2 α N γ + α S N + γ γ + α .

Then we get

(4.34) K = γ + α 2 ( N + γ ) μ 1 α N γ + α + μ 2 α N γ + α S N + γ γ + α .

If K is attained by (u, v) N, then by Lemma 4.4 we know that (u, v) is a nontrivial solution of (1.1). Thus we get Nϕu|u|p>0. Hence we infer that

(4.35) | u | D 1 , α 2 < μ 1 R N R N | u ( x ) | p | u ( y ) | p | x y | N γ d x d y μ 1 S p | u | D 1 , α 2 p ,  and  | v | D 1 , α 2 < μ 2 R N R N | v ( x ) | p | v ( y ) | p | x y | N γ d x d y μ 2 S p | v | D 1 , α 2 p .

Then we infer from 1.20 that

(4.36) K = J ( u , v ) = γ + α 2 ( N + γ ) | u | D 1 , α 2 + | v | D 1 , α 2 > γ + α 2 ( N + γ ) μ 1 α N γ + α + μ 2 α N γ + α S N + γ γ + α .

This is a contradiction. Then we finish the proof.

Next we prove that the system (1.1) has a ground state solution for each β > 0. To this purpose we define

(4.37) K1:=inf(u,v)N1(u,v),

where

(4.38) N 1 = { ( u , v ) D α ( 0 , 0 ) : J ( u , v ) ( u , v ) = 0 } .

It is clear that N N1 and K1K. Furthermore, one can check that K1 > 0. In order to study the existence of ground state solution, we shall consider the system (1.1) in the bounded domain BR(0) = {x ∈N : |x| < R}. Set DRα:=Dα(0,R)×Dα(0,R), where Dα(0, R) is similar to 1.14-(1.17) when ℝN is replaced by BR(0). Moreover, we require u = 0 if u ∈ ∂BR(0). Next we consider the system

(4.39) ( Δ ) α 2 u = μ 1 ϕ u , R | u | p 2 u + β ϕ v , R | u | p 2 u , x B R ( 0 ) , ( Δ ) α 2 v = μ 2 ϕ v , R | v | p 2 v + β ϕ u , R | v | p 2 v , x B R ( 0 ) , ( u , v ) D R α ,

where ϕu,R(x)=BR(0)|u(y)|p|xy|Nγdy and p=N+γNα. We define

(4.40) K ( R ) := inf ( u , v ) N ( R ) J ( u , v ) ,

where

(4.41) N ( R ) = { ( u , v ) D R α ( 0 , 0 ) : G ( u , v ) = J ( u , v ) ( u , v ) = 0 } .

The next lemma proves K(R) = K1 for each R > 0.

Lemma 4.5

The conclusion K(R) = K1 holds for each R > 0.

Proof. We first prove that K(R1) = K(R2) for each R1 > R2. In fact, it is easy to see that N(R2) N(R1). This implies K(R1)≤ K(R2). Next we shall prove the reverse inequality. For each (u1, v1) N(R1) we set

(4.42) ( u 2 ( x ) , v 2 ( x ) ) = R 1 R 2 N α 2 u 1 R 1 R 2 x , v 1 R 1 R 2 x .

Then it follows that (u2, v2) N(R2) and

(4.43) K ( R 2 ) J ( u 2 , v 2 ) = J ( u 1 , v 1 ) for each ( u 1 , v 1 ) N ( R 1 ) .

That is, we obtain K(R2)≤ K(R2) and K(R1) = K(R2). Next we prove K(R) = K1. It is easy to see that K1K(R). Let {(un , vn)} ⊂ N1 be the minimizing sequence of K1. Without loss of generality we can assume that (un,vn)DRnα for some Rn →∞, as n → ∞. Then we have (un , vn) N(Rn) and

(4.44) K 1 = lim n J ( u n , v n ) lim n K ( R n ) = K ( R ) .

This gives the conclusion K(R) = K1 for each R > 0.

In order to overcome the lack of compactness we consider the following perturbation problem

(4.45) ( Δ ) a 2 u = μ 1 ϕ u , R , ϵ | u | p 2 ϵ u + β ϕ v , R , ϵ | u | p 2 ϵ u , x B R ( 0 ) , ( Δ ) a 2 v = μ 2 ϕ v , R | v | p 2 ϵ v + β ϕ u , R , ϵ | v | p 2 ϵ v , x B R ( 0 ) , ( u , v ) D R α ,

where ϕu,R(x)=BR(0)|u(y)|pϵ|xy|Nγdy and 0<ϵ<p1, Correspondingly, we define

(4.46) Kϵ=inf(u,v)NϵJϵ(u,v),

where

(4.47) J ϵ ( u , v ) = 1 2 | u | D α ( 0 , R ) 2 + | v | D α ( 0 , R ) 2 1 2 p 2 ϵ R N μ 1 ϕ u , R , ϵ | u | p ϵ + μ 2 ϕ v , R , ϵ | v | p ϵ + 2 β ϕ u , R , ϵ | v | p ϵ , N ϵ = ( u , v ) D R α ( 0 , 0 ) : J ϵ ( u , v ) ( u , v ) = 0 .

In the next lemma we follow the idea of [10, 50] to give the estimates for Kϵ.

Lemma 4.6

For each ϵ < p − 1, we have

(4.48) K ϵ < min inf ( u , 0 ) N ϵ ϵ ( u , 0 ) , inf ( 0 , v ) N ϵ J ϵ ( 0 , v ) .

Proof. For any 0 < ϵ < p − 1, we know that the equation

(4.49) ( Δ ) α 2 u = μ i ϕ u , R , ϵ | u | p 2 ϵ u , u D α ( 0 , R )

has a least energy solution ui(i = 1, 2). Hence we know that

(4.50) J ϵ u 1 , 0 = d 1 = inf ( u , 0 ) N ε J ϵ ( u , 0 )  and  J ϵ 0 , u 2 = d 2 = inf ( 0 , v ) N ε J ϵ ( 0 , v ) .

We divide into the following two cases to prove our results. If 1 + ϵ < p ≤ 2, the conclusion follows from [10, Lemma 2.7]. If p > 2, we define

(4.51) ȷ ~ ϵ ( u , v ) = | u | D α ( 0 , R ) 2 + | u | D α ( 0 , R ) 2 2 R N μ 1 ϕ u , R , ϵ | u | p ϵ + μ 2 ϕ v , R , ϵ | v | p ϵ + 2 β ϕ u , R , ϵ | v | p ϵ .

As in [50, Lemma 3.3], we know that

(4.52) K ϵ = inf ( u , v ) D R α { ( 0 , 0 ) } J ~ ϵ ( u , v ) .

Now we consider the function

(4.53) f ( s , t ) = J ~ ε s u 1 , t u 2 = s 2 u 1 D α ( 0 , R ) 2 + t 2 u 2 D α ( 0 , R ) 2 2 R N μ 1 s 2 p ϕ u 1 , R , ϵ u 1 p ϵ + μ 2 t 2 p ϕ u 2 , R , ϵ u 2 p ϵ + 2 s p s p β ϕ u 1 , R , ϵ u 2 p ϵ ,

where (s, t) ∈ Λ = {(s, t) : s ≥ 0, t ≥ 0, (s, t) ≠ (0, 0)}. Then it is sufficient to show that f does not attain its minimum over Λ on the lines s = 0 or t = 0. We infer from p > 2 that

(4.54) f S ( s , 0 ) < 0 , f s ( 0 , t ) = 0 , f t ( 0 , t ) < 0  and  f t ( s , 0 ) = 0.

Hence we know that f can not attain its minimum over Λ on the lines s = 0 or t = 0. This finishes the proof.

Recall that (wμ1,wμ2)=(μ1αN2(γ+α)U1(x),μ2aN2(γ+α)U1(x)). By using the same arguments as in Lemma 4.6, we deduce that

(4.55) K 1 < min inf ( u , 0 ) N 1 ( u , 0 ) , inf ( 0 , v ) N 1 J ( 0 , v ) = min J w μ 1 , 0 , J 0 , w μ 2 = min γ + α 2 ( N + γ ) μ 1 α N γ α S N + γ γ + α , γ + α 2 ( N + γ ) μ 2 α N γ + α S N + γ γ + α .

Next we prove the existence of nontrivial solution of (4.45).

Lemma 4.7

For each ϵ < p − 1, we know that (4.45) has a classical least energy solution (uϵ , vϵ).

Proof. Let {(un , vn)} ⊂ Nϵ be the minimizing sequence of Kϵ. That is, Jϵ(un , vn)→ Kϵ as n → ∞. Hence we obtain that

(4.56) K ϵ = lim n γ ϵ u n , v n = lim n 1 2 1 2 p 2 ϵ u n D α ( 0 , R ) 2 + v n D α ( 0 , R ) 2 .

Then {(un , vn)} is bounded in DRα. Passing to a subsequence, we may assume that un uϵ , vn vϵ weakly in Dα(0, R). By the compactness of the embedding Dα(0,R)L2p2ϵ(BR(0)) see [15, Corrollary 7.2]), we infer from Hardy-Littlewood-Sobolev inequality(see Lemma 2.1) that

(4.57) 2 p 2 ϵ p 1 ϵ K ϵ = 2 p 2 ϵ p 1 ϵ lim n J ϵ u n , v n = lim n B R ( 0 ) μ 1 ϕ u n , R , ϵ u n p ϵ + μ 2 ϕ v n , R , ϵ v n p ϵ + 2 β ϕ u n , R , ϵ v n p ϵ = B R ( 0 ) μ 1 ϕ u ε , R , ϵ u ϵ p ϵ + μ 2 ϕ V ε , R , ϵ v ϵ p ϵ + 2 β ϕ u ε , R , ϵ v ϵ p ϵ .

This implies that (uϵ , vϵ) ≠(0, 0). Furthermore, we infer from Fatou’s lemma that

(4.58) u ϵ D α ( 0 , R ) 2 + v ϵ D α ( 0 , R ) 2 lim n u n D α ( 0 , R ) 2 + v n D α ( 0 , R ) 2 = B R ( 0 ) μ 1 ϕ u ϵ , R , ϵ u ϵ p ϵ + μ 2 ϕ v ε , R , ϵ v ϵ p ϵ + 2 β ϕ u ε , R , ϵ v ϵ p ϵ .

Therefore, it is easy to check that there exists 0 < tϵ ≤ 1 such that (tϵuϵ , tϵvϵ) Nϵ. Then we infer that

(4.59) K ϵ J ϵ t ϵ u ϵ , t ϵ v ϵ = p 1 ϵ 2 p 2 ϵ t ϵ 2 u ϵ D α ( 0 , R ) 2 + v ϵ D α ( 0 , R ) 2 lim n u n D α ( 0 , R ) 2 + v n D α ( 0 , R ) 2 = lim n ϵ u n , v n = K ϵ .

Then we get that tϵ = 1 and Jϵ(uϵ , vϵ) = Kϵ. Furthermore, we have

(4.60) lim n u n D α ( 0 , R ) 2 + v n D α ( 0 , R ) 2 = u ϵ D α ( 0 , R ) 2 + v ϵ D α ( 0 , R ) 2 .

That is, (un , vn)→ (uϵ , vϵ) in DRα. As in [53, Lemma 2.3], one can prove that the Nehari type constraint set N(R) is a natural constraint. That is, (uϵ , vϵ) is a solution of (4.45). Furthermore, we infer from Lemma 4.6 that uϵ ≠0 and vϵ ≠0. By regularity theory(see [25]), we see that uϵ , vϵ ∈ Cα(BR(0)).

Now we are ready to give the proof Theorem 1.4 (ii).

Proof of Theorem 1.4 (ii). It is clear that for each (u, v) N(1), there exists tϵ > 0 such that (tϵu, tϵv) Nϵ and tϵ → 1 as ϵ → 0. Hence we infer that

(4.61) lim sup ϵ 0 K ϵ lim sup ϵ 0 ϵ t ϵ u , t ϵ v = J ( u , v ) .

Then we infer from Lemma 4.5 that

(4.62) limsupϵ0KϵK(1)=K1.

Let (uϵ , vϵ) be the least energy solution of (1.1). Then we have J′ϵ(uϵ , vϵ)(uϵ , vϵ) = 0. From Hardy-Littlewood-Sobolev inequality and Sobolev inequality, we deduce that

(4.63) 2 p 2 ϵ p 1 ϵ K ϵ = u ϵ D α ( 0 , R ) 2 + v ϵ D α ( 0 , R ) 2 σ 0  for each  0 < ϵ < p 1 4 ,

where σ0 is a positive constant independent of ϵ. Then we know that (uϵ , vϵ) is uniformly bounded in Dα(0, R). Passing to a subsequence, wemay assume that uϵ u and vϵ v weakly in Dα(0, R). Then (u, v) is a solution of

(4.64) ( Δ ) α 2 u = μ 1 ϕ u , 1 | u | p 2 u + β ϕ v , 1 | u | p 2 u , x B 1 ( 0 ) , ( Δ ) α 2 v = μ 2 ϕ v , 1 | v | p 2 v + β ϕ u , 1 | v | p 2 v , x B 1 ( 0 ) , ( u , v ) D 1 α .

Next we divide into the following two cases to prove the results. That is, |uϵ| + |vϵ| is bounded, or |uϵ| + |vϵ| →∞as ϵ → 0.

If |uϵ| + |vϵ| is bounded, we infer from the Dominated Convergence Theorem that

(4.65) lim ϵ 0 B 1 ( 0 ) ϕ u ε , 1 , ϵ u ϵ p ϵ = B 1 ( 0 ) ϕ u , 1 | u | p , lim ϵ 0 B 1 ( 0 ) ϕ v ε , 1 , ϵ v ϵ p ϵ = B 1 ( 0 ) ϕ v , 1 | v | p ,  and  lim ϵ 0 B 1 ( 0 ) ϕ u ε , 1 , ϵ v ϵ p ϵ = B 1 ( 0 ) ϕ u , 1 | v | p .

Moreover, since J′ϵ(uϵ , vϵ) = J′(u, v) = 0, it follows that uϵu and vϵv in Dα(0, R). Moreover, we infer from (4.63) that (u, v) ≠(0, 0). Without loss of generality we assume that u ≠0. From [41, Proposition 3.1] and [45, Proposition 1.6], we infer that the system (4.39) has following Pohozaev identity

(4.66) α N 2 B 1 ( 0 ) μ 1 ϕ u , 1 | u | p + μ 2 ϕ u , 1 | u | p + 2 β ϕ u , 1 | u | p Γ α + 2 2 2 B 1 ( 0 ) u δ 2 + v δ 2 x v d σ = N + γ 2 p B 1 ( 0 ) μ 1 ϕ u , 1 | u | p + μ 2 ϕ u , 1 | u | p + 2 β ϕ u , 1 | u | p ,

where δ(s) = dist(x, ∂B1(0)), ν is the unit outward normal vector to ∂BR(0) at x, and Γ is the Gamma function. Hence we infer that

(4.67) 0 < Γ α + 2 2 2 B 1 ( 0 ) u δ 2 + v δ 2 x v d σ = 0.

This is a contradiction. Hence we have |uϵ| + |vϵ| →∞as ϵ → 0. In the following we shall use the blowup arguments. To this purpose we denote λϵ={ | uϵ |,| vϵ | }. Then λϵ →∞as ϵ → 0. We define

(4.68) U ϵ ( x ) = λ 1 u ϵ λ χ ϵ X  and  V ϵ ( x ) = λ 1 u ϵ λ χ ϵ x ,  where  χ ϵ = 2 ( p 1 ϵ ) γ + α .

Thus, we have that

(4.69) 1=max{ | Uϵ |,| Vϵ | }

and (Uϵ(x), Vϵ(x)) satisfies

(4.70) ( Δ ) α 2 U ϵ = μ 1 ϕ U ϵ , R , ϵ U ϵ p 2 ϵ u + β ϕ v ϵ , R , ϵ | u | p 2 ϵ u , x B λ ε χ ( 0 ) , ( Δ ) α 2 V ϵ = μ 2 ϕ V ϵ , R , ϵ V ϵ p 2 ϵ V ϵ + β ϕ U ϵ , R , ϵ V ϵ p 2 ϵ V ϵ , x B λ ϵ χ ϵ ( 0 ) .

A direct computation shows that

(4.71) U ϵ D 1 , α 2 = λ ϵ ( N α ) ϵ u ϵ D 1 , α 2 u ϵ D 1 , α 2 .

Thus, we infer from (4.62)-(4.63) that (Uϵ , Vϵ) is bounded in Dα. By regularity results for fractional nonlocal problems(see [25]), for a subsequence we have UϵU and VϵV in Clocα as ϵ0. Moreover, since |Uϵ|, |Vϵ| ≤ 1, it follows that ϕUϵ,R,ϵ(x)ϕU,R(x) and ϕVϵ,R,ϵ(x)ϕV,R(x) in Clocα as ϵ0. Thus we know that (U, V) satisfies (1.1). From (4.69) we infer that (U, V) ≠(0, 0) and (U, V) N1. Then we infer from (4.61) that

(4.72) K 1 J ( U , V ) = p 1 2 p | U | D 1 , α 2 + | U | D 1 , α 2 lim inf ϵ 0 p 1 ϵ 2 p 2 ϵ U n D α 0 , λ ε χ 2 + V n D α 0 , λ ε χ ϵ 2 lim inf ϵ 0 p 1 ϵ 2 p 2 ϵ u n D α ( 0 , 1 ) 2 + u n D α ( 0 , 1 ) 2 = lim inf ϵ 0 K ϵ K 1 .

This together with (4.55) imply that K1 = J(U, V) and U, V ≠0. Also, we know that (U, V) N and J(U, V) ≥ K. Thus, we have K1 = J(U, V) = K and (U, V) is nontrivial ground state solution of (1.1).

Next we find the unique solution of (1.21) for |β| small.

Lemma 4.8

There exists β˜0>0 small such that (k(β), l(β)) is the unique solution of (1.21) for 0<β<β˜0. Moreover, we have

limβ0(k2(β)+l2(β))=k2(0)+l2(0)=μ1Nαγ+α+μ2Nαγ+α.

Proof. We shall use the implicit function theorem to prove the existence of (k(β), l(β)) for β > 0 small. For convenience we let gi(k, l, β) to denote gi(k, l)(i = 1, 2). We define k(0)=μ112(p1) and l(0)=μ212(p1). Thus we have gi(k(0), l(0), 0) = 0(i = 1, 2). Moreover, a direct computation shows that

(4.73) k g 1 ( k ( 0 ) , l ( 0 ) , 0 ) = 2 ( p 1 ) μ 1 k ( 0 ) 2 p 3 > 0 , l g 2 ( k ( 0 ) , l ( 0 ) , 0 ) = 2 ( p 1 ) μ 2 l ( 0 ) 2 p 3 > 0 , l g 1 ( k ( 0 ) , l ( 0 ) , 0 ) = k g 2 ( k ( 0 ) , l ( 0 ) , 0 ) = 0.

This implies that

(4.74) det k g 1 ( k ( 0 ) , l ( 0 ) , 0 ) l g 1 ( k ( 0 ) , l ( 0 ) , 0 ) k g 2 ( k ( 0 ) , l ( 0 ) , 0 ) l g 2 ( k ( 0 ) , l ( 0 ) , 0 ) > 0.

Then we apply the implicit function theorem to obtain that k(β), l(β) are well defined and of class C1 on (β˜0,β˜0) for some β˜0>0, and gi(k(β),l(β),β)=0(i=1,2). Hence we know that (k(β)U1,l(β)U1) is a positivesolution of (1.1). Furthermore, we get

(4.75) lim β 0 k 2 ( β ) + l 2 ( β ) = k 2 ( 0 ) + l 2 ( 0 ) = μ 1 N α γ + α + μ 2 N α γ + α .

Thus, there exists 0<β0<β˜0 such that

(4.76) k 2 ( β ) + l 2 ( β ) > min μ 1 N α γ + α , μ 2 N α γ + α  for  β 0 , β 0 .

We infer from (4.79) and (4.55) that

(4.77) J ( U , V ) = K 1 = K < J k ( β ) U 1 , l ( β ) U 1 , β 0 , β 0 .

This implies that (k(β)U1,l(β)U1) is different positive solution of (1.1) from (U, V). This completes the proof.

Then from Lemma 4.8 we know that Theorem 1.4 (iii) holds. Finally, we prove the Theorem 1.4 (iv).

Proof of Theorem 1.4 (iv). We first consider the case 2α + γ > N. It is clear that (k0(β), l0(β)) = (k(β), l(β)) for β > 0 small. From (4.2) we infer that k(β), l(β) are strictly decreasing for β > 0. Hence we know that there exists β1 > β0 such that

(4.78) k 2 ( β ) + l 2 ( β ) < min μ 1 N α γ + α , μ 2 N α γ + α  for  β β 1 , + .

Next we prove that the solution (k0U1, l0U1) is a ground state solution of (1.1) for β > β1. In fact, since (k0U1, l0U1) is a nontrivial solution of (1.1), it follows that

(4.79) K J k 0 U 1 , l 0 U 1 = γ + α 2 ( N + γ ) k 0 2 + l 0 2 S N + γ γ + α .

Let {(un , vn)} be a minimizing sequence for K. Then we infer from (1.16) that

(4.80) S e n u n D 1 , α 2 = μ 1 R N ϕ u n u n p + β R N ϕ u n v n p μ 1 e n p + β e n p 2 f n p 2 , S f n v n D 1 , α 2 = μ 2 R N ϕ v n v n p + β R N ϕ u n v n p μ 2 f n p + β e n p 2 f n p 2 ,

where

(4.81) e n = R N ϕ u n u n p 1 p  and  f n = R N ϕ v n v n p 1 p .

This implies that

(4.82) S e n + f n 2 ( N + γ ) γ + α J u n , v n k 0 2 + l 0 2 S N + γ γ + α , S μ 1 e n p 1 + β e n p 2 1 f n p 2 , S μ 2 f n p 1 + β f n p 2 1 e n p 2 .

The first inequality implies that en , fn are bounded. Then we can assume that ene, fnf, as n →∞.From (1.20), we deduce that μ1ep+2βep2fp2+μ2fp2(N+γ)γ+αK>0. Thus, at least one of e, f is nonzero. Assume that e > 0 and f = 0. Since 2α + γ > N, we know that p > 2. Hence we infer from the third inequality of (4.82) that S ≤ 0. This is a contradiction. Thus, we have e > 0 and f > 0. Set k=(e/SNαγ+α)12 and l=(f/SNαγ+α)12. Let n → ∞, we know that (4.82) becomes (4.8). This implies that enk02SNαγ+α and fnl02SNαγ+α, as n. We infer from (4.82) that

(4.83) lim n J u n , v n k 0 2 + l 0 2 S N + γ γ + α .

Combining (4.79) and (4.83), we know that

(4.84) K = γ + α 2 ( N + γ ) k 0 2 + l 0 2 S N + γ γ + α .

This finishes the proof of the case 2α+γ>N and 0<βμ1μ2.

For the case 2α + γ = N(i.e., p = 2) and β ∈ (0, min1, μ 2}) (max1, μ2},∞), by using the same arguments as in Lemma 4.2 we know that (μ2βμ1μ2β2,μ1βμ1μ2β2) is the unique solution of (1.21). Hence as in the above one can prove that (μ2βμ1μ2β2U1,μ1βμ1μ2β2U1) is a ground state solution of (1.1). Similarly, one can prove the conclusions for the case 2α + γ < N and β ≥ (p − 1) max1, μ2}.

Acknowledgement

The author thanks the anonymous referees for his/her very valuable comments, which led to an improvement of this paper. This work was supported by NSF of China (Grants 11971202, 11671077), the Outstanding Young foundation of Jiangsu Province(Grant BK20200042) and the Six big talent peaks project in Jiangsu Province(XYDXX-015).

  1. Conflict of interest: Authors state no conflict of interest

References

[1] S. Abe and A. Ogura. Solitary waves and their critical behavior in a nonlinear nonlocal medium with power-law response. Phys. Rev. E, 57(5):6066–6070, 1998.10.1103/PhysRevE.57.6066Search in Google Scholar

[2] Luis A. Caffarelli, B. Gidas, and J. Spruck. Asymptotic symmetry and local behavior of semilinear elliptic equations with critical Sobolev growth. Comm. Pure Appl. Math., 42(3):271–297, 1989.10.1002/cpa.3160420304Search in Google Scholar

[3] W.-X. Chen, Y.-Q. Fang, and R. Yang. Liouville theorems involving the fractional Laplacian on a half space. Adv. Math., 274:167–198, 2015.10.1016/j.aim.2014.12.013Search in Google Scholar

[4] W.-X. Chen and C.-M. Li. Classification of solutions of some nonlinear elliptic equations. Duke Math. J., 63(3):615–622, 1991.10.1215/S0012-7094-91-06325-8Search in Google Scholar

[5] W.-X. Chen and C.-M. Li. Methods on nonlinear elliptic equations, volume 4 of AIMS Series on Differential Equations & Dynamical Systems. American Institute of Mathematical Sciences (AIMS), Springfield, MO, 2010.Search in Google Scholar

[6] W.-X. Chen and C.-M. Li. Maximum principles for the fractional p-Laplacian and symmetry of solutions. Adv. Math., 335:735–758, 2018.10.1016/j.aim.2018.07.016Search in Google Scholar

[7] W.-X. Chen, C.-M. Li, and Y. Li. A direct method of moving planes for the fractional Laplacian. Adv. Math., 308:404–437, 2017.10.1016/j.aim.2016.11.038Search in Google Scholar

[8] W.-X. Chen, C.-M. Li, and B. Ou. Classification of solutions for an integral equation. Comm. Pure Appl. Math., 59(3):330–343, 2006.10.1002/cpa.20116Search in Google Scholar

[9] W.-X. Chen, Y. Li, and R.-B. Zhang. A direct method of moving spheres on fractional order equations. J. Funct. Anal., 272(10):4131–4157, 2017.10.1016/j.jfa.2017.02.022Search in Google Scholar

[10] Z.-J. Chen and W.-M. Zou. Positive least energy solutions and phase separation for coupled Schrödinger equations with critical exponent: higher dimensional case. Calc. Var. Partial Differential Equations, 52(1-2):423–467, 2015.10.1007/s00526-014-0717-xSearch in Google Scholar

[11] W. Dai, Y.-Q. Fang, and G.-L. Qin. Classification of positive solutions to fractional order Hartree equations via a direct method of moving planes. J. Differential Equations, 265(5):2044–2063, 2018.10.1016/j.jde.2018.04.026Search in Google Scholar

[12] W. Dai, J.-H. Huang, Y. Qin, B. Wang, and Y.-Q. Fang. Regularity and classification of solutions to static Hartree equations involving fractional Laplacians. Discrete Contin. Dyn. Syst., 39(3):1389–1403, 2019.10.3934/dcds.2018117Search in Google Scholar

[13] W. Dai and G.-L. Qin. Classification of nonnegative classical solutions to third-order equations. Adv. Math., 328:822–857, 2018.10.1016/j.aim.2018.02.016Search in Google Scholar

[14] F. Dalfovo, S. Giorgini, L. P. Pitaevskii, and S. Stringari. Theory of bose-einstein condensation in trapped gases. Rev. Mod. Phys., 71(3):463–512, 1999.10.1103/RevModPhys.71.463Search in Google Scholar

[15] E. Di Nezza, G. Palatucci, and E. Valdinoci. Hitchhiker’s guide to the fractional Sobolev spaces. Bull. Sci. Math., 136(5):521–573, 2012.10.1016/j.bulsci.2011.12.004Search in Google Scholar

[16] Y. Fang and W. Chen. A Liouville type theorem for poly-harmonic Dirichlet problems in a half space. Adv. Math., 229(5):2835–2867, 2012.10.1016/j.aim.2012.01.018Search in Google Scholar

[17] J. Fröhlich, T.-P. Tsai, and H.-T. Yau. On a classical limit of quantum theory and the non-linear Hartree equation. Geom. Funct. Anal., (Special Volume, Part I):57–78, 2000. GAFA 2000 (Tel Aviv, 1999).10.1007/978-3-0346-0422-2_3Search in Google Scholar

[18] J. Fröhlich, T.-P. Tsai, and H.-T. Yau. On the point-particle (Newtonian) limit of the non-linear Hartree equation. Comm. Math. Phys., 225(2):223–274, 2002.10.1007/s002200100579Search in Google Scholar

[19] V. Georgiev and G. Venkov. Symmetry and uniqueness of minimizers of Hartree type equations with external Coulomb potential. J. Differential Equations, 251(2):420–438, 2011.10.1016/j.jde.2011.04.012Search in Google Scholar

[20] M. Ghimenti and J. Van Schaftingen. Nodal solutions for the Choquard equation. J. Funct. Anal., 271(1):107–135, 2016.10.1016/j.jfa.2016.04.019Search in Google Scholar

[21] B. Gidas, W.-M. Ni, and L. Nirenberg. Symmetry and related properties via the maximum principle. Comm. Math. Phys., 68(3):209–243, 1979.10.1007/BF01221125Search in Google Scholar

[22] B. Gidas, W.-M. Ni, and L. Nirenberg. Symmetry of positive solutions of nonlinear elliptic equations in Rn. In Mathematical analysis and applications, Part A, volume 7 of Adv. in Math. Suppl. Stud., pages 369–402. Academic Press, New York-London, 1981.Search in Google Scholar

[23] J. Ginibre and G. Velo. The classical field limit of scattering theory for nonrelativistic many-boson systems. I. Comm. Math. Phys., 66(1):37–76, 1979.10.1007/BF01197745Search in Google Scholar

[24] K. Hepp. The classical limit for quantum mechanical correlation functions. Comm. Math. Phys., 35:265–277, 1974.10.1007/BF01646348Search in Google Scholar

[25] D. Goel J. Giacomoni and K. Sreenadh. Regularity results on a class of doubly nonlocal problems. Preprint, arXiv:1909.10648v1, 2019.10.1016/j.jde.2019.11.009Search in Google Scholar

[26] W. Krolikowski, O. Bang, N. I. Nikolov, D. Neshev, J. Wyller, J. J. Rasmussen, and D. Edmundson. Modulational instability, solitons and beam propagation in spatially nonlocal nonlinear media. J. Opt. B: Quantum Semiclass. Opt., 6(5):S288–S294, 2004.10.1088/1464-4266/6/5/017Search in Google Scholar

[27] T. Kulczycki. Properties of Green function of symmetric stable processes. Probab. Math. Statist., 17(2, Acta Univ. Wratislav. No. 2029):339–364, 1997.Search in Google Scholar

[28] P. Le. Liouville theorem and classification of positive solutions for a fractional Choquard type equation. Nonlinear Anal., 185:123–141, 2019.10.1016/j.na.2019.03.006Search in Google Scholar

[29] C. Le Bris and P.-L. Lions. From atoms to crystals: a mathematical journey. Bull. Amer. Math. Soc. (N.S.), 42(3):291–363, 2005.10.1090/S0273-0979-05-01059-1Search in Google Scholar

[30] Y.-T. Lei. On the regularity of positive solutions of a class of Choquard type equations. Math. Z., 273(3-4):883–905, 2013.10.1007/s00209-012-1036-6Search in Google Scholar

[31] Y.-T. Lei. Qualitative analysis for the static Hartree-type equations. SIAM J. Math. Anal., 45(1):388–406, 2013.10.1137/120879282Search in Google Scholar

[32] Y.-Y. Li and M.-J. Zhu. Uniqueness theorems through the method of moving spheres. Duke Math. J., 80(2):383–417, 1995.10.1215/S0012-7094-95-08016-8Search in Google Scholar

[33] E.-H. Lieb. Existence and uniqueness of the minimizing solution of Choquard’s nonlinear equation. Studies in Appl. Math., 57(2):93–105, 1976/77.10.1007/978-3-642-55925-9_37Search in Google Scholar

[34] E.-H. Lieb and M. Loss. Analysis, volume 14 of Graduate Studies in Mathematics. American Mathematical Society, Providence, RI, second edition, 2001.10.1090/gsm/014Search in Google Scholar

[35] E.-H. Lieb and B. Simon. The Hartree-Fock theory for Coulomb systems. Comm. Math. Phys., 53(3):185–194, 1977.10.1007/BF01609845Search in Google Scholar

[36] C.-S. Lin. A classification of solutions of a conformally invariant fourth order equation in Rn Comment. Math. Helv., 73(2):206–231, 1998.10.1007/s000140050052Search in Google Scholar

[37] P.-L. Lions. The Choquard equation and related questions. Nonlinear Anal., 4(6):1063–1072, 1980.10.1016/0362-546X(80)90016-4Search in Google Scholar

[38] P.-L. Lions. Some remarks on Hartree equation. Nonlinear Anal., 5(11):1245–1256, 1981.10.1016/0362-546X(81)90016-XSearch in Google Scholar

[39] P.-L. Lions. Solutions of Hartree-Fock equations for Coulomb systems. Comm. Math. Phys., 109(1):33–97, 1987.10.1007/BF01205672Search in Google Scholar

[40] L. Ma and L. Zhao. Classification of positive solitary solutions of the nonlinear Choquard equation. Arch. Ration. Mech. Anal., 195(2):455–467, 2010.10.1007/s00205-008-0208-3Search in Google Scholar

[41] V. Moroz and J. Van Schaftingen. Groundstates of nonlinear Choquard equations: existence, qualitative properties and decay asymptotics. J. Funct. Anal., 265(2):153–184, 2013.10.1016/j.jfa.2013.04.007Search in Google Scholar

[42] V. Moroz and J. Van Schaftingen. Existence of groundstates for a class of nonlinear Choquard equations. Trans. Amer. Math. Soc., 367(9):6557–6579, 2015.10.1090/S0002-9947-2014-06289-2Search in Google Scholar

[43] B. Ou. A remark on a singular integral equation. Houston J. Math., 25(1):181–184, 1999.Search in Google Scholar

[44] P. Pedri and L. Santos. Two-dimensional bright solitons in dipolar bose-einstein condensates. Phys. Rev. Lett., 95:200404, Nov 2005.10.1109/EQEC.2005.1567400Search in Google Scholar

[45] X. Ros-Oton and J. Serra. The Pohozaev identity for the fractional Laplacian. Arch. Ration. Mech. Anal., 213(2):587–628, 2014.10.1007/s00205-014-0740-2Search in Google Scholar

[46] D. Ruiz and J. Van Schaftingen. Odd symmetry of least energy nodal solutions for the Choquard equation. J. Differential Equations, 264(2):1231–1262, 2018.10.1016/j.jde.2017.09.034Search in Google Scholar

[47] L. Santos, G. V. Shlyapnikov, P. Zoller, and M. Lewenstein. Bose-einstein condensation in trapped dipolar gases. Phys. Rev. Lett., 85:1791–1794, Aug 2000.10.1103/PhysRevLett.85.1791Search in Google Scholar PubMed

[48] J. Serrin. A symmetry problem in potential theory. Arch. Rational Mech. Anal., 43:304–318, 1971.10.1007/BF00250468Search in Google Scholar

[49] L. Silvestre. Regularity of the obstacle problem for a fractional power of the Laplace operator. Comm. Pure Appl. Math., 60(1):67–112, 2007.10.1002/cpa.20153Search in Google Scholar

[50] B. Sirakov. Least energy solitary waves for a system of nonlinear Schrödinger equations in Rn Comm. Math. Phys., 271(1):199–221, 2007.10.1007/s00220-006-0179-xSearch in Google Scholar

[51] J. Wang. Existence of normalized solutions for the coupled hartree-fock type system. Preprint, 2018.10.1002/mana.201900230Search in Google Scholar

[52] J. Wang and Y.-Y. Dong. Bonded states solutions for a coupled nonlinear hartree equations with nonlocal interaction. Preprint, 2017.10.1007/s00526-017-1268-8Search in Google Scholar

[53] J. Wang and J.-P. Shi. Standing waves for a coupled nonlinear Hartree equations with nonlocal interaction. Calc. Var. Partial Differential Equations, 56(6):56:168, 2017.10.1007/s00526-017-1268-8Search in Google Scholar

[54] J. Wang and W. Yang. Normalized solutions and asymptotical behavior of minimizer for the coupled Hartree equations. J. Differential Equations, 265(2):501–544, 2018.10.1016/j.jde.2018.03.003Search in Google Scholar

[55] J.-C. Wei and X.-W. Xu. Classification of solutions of higher order conformally invariant equations. Math. Ann., 313(2):207–228, 1999.10.1007/s002080050258Search in Google Scholar

[56] M.-B. Yang, Y.-H. Wei, and Y.-H. Ding. Existence of semiclassical states for a coupled Schrödinger system with potentials and nonlocal nonlinearities. Z. Angew. Math. Phys., 65(1):41–68, 2014.10.1007/s00033-013-0317-1Search in Google Scholar

[57] R. Zhuo, W. Chen, X. Cui, and Z. Yuan. A liouville theorem for the fractional laplacian. Preprint, arXiv:1401.7402, 2014.Search in Google Scholar

Received: 2021-01-09
Accepted: 2021-07-19
Published Online: 2021-08-26

© 2021 Jun Wang, published by De Gruyter

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 8.6.2024 from https://www.degruyter.com/document/doi/10.1515/anona-2021-0202/html
Scroll to top button