Introduction

Consider the following convex problem: (1) where f : ℝⁿ → ℝ is a possibly nonsmooth convex function. In general, this problem has been well studied for several decades when f is continuously differentiable, and a number of different methods have been developed for its solution Eq (1) (for example, numerical optimization method [1–3] etc, heuristic algorithm [4–6] etc). However, when f is a nondifferentiable function, the difficulty of solving this problem increases. Recently, such problems have arisen in many medical, image restoration and optimal control applications (see [7–13] etc). Some authors have previously studied nonsmooth convex problems (see [14–18] etc).

Let F : ℝⁿ → ℝ be the so-called Moreau-Yosida regularization of f, which is defined by (2) where λ is a positive parameter and ‖ ⋅ ‖ denotes the Euclidean norm. The problem Eq (1) is equivalent to the following problem (3) It is well known that the problems Eqs (1) and (3) of the solution sets are the same. As we know, one of the most effective methods for problems Eq (3) is the trust region method.

The trust region method plays an important role in the area of nonlinear optimization, and it has been proven to be a very efficient method. Levenberg [19] and Marquardt [20] first applied this method to nonlinear least-squares problems, and Powell [21] established a convergence result for this method for unconstrained problems. Fletcher [22] first proposed a trust region method for composite nondifferentiable optimization problems. Over the past decades, many authors have studied the trust region algorithm to minimize nonsmooth objective function problems. For example, Sampaio, Yuan and Sun [23] used the trust region algorithm for nonsmooth optimization problems; Sun, Sampaio and Yuan [24] proposed a quasi-Newton trust region algorithm for nonsmooth least-squares problems; Zhang [25] used a new trust region algorithm for nonsmooth convex minimization; and Yuan, Wei and Wang [26] proposed a gradient trust region algorithm with a limited memory BFGS update for nonsmooth convex minimization problems. For other references on trust region methods, see [27–35], among others. In particular, for the problem we address in this study, as we can compute the exact Hessian, the trust region method could be very efficient. However, it is difficult to compute the Hessian at every iteration, which increases the computational workload and time.

The purpose of this paper is to present an efficient trust region algorithm to solve Eq (3). With the use of the Moreau-Yosida regularization (smoothing) and the new quasi-Newton equation, the given method has the following good properties: (i) the Hessian makes use of not only the gradient value but also the function value and (ii) the subproblem of the proposed method, which possesses the form of an unconstrained trust region subproblem, can be solved using existing methods.

The remainder of this paper is organized as follows. In the next section, we briefly review some basic results in convex analysis and nonsmooth analysis and state a new quasi-Newton secant equation. In section 3, we present a new algorithm for solving problem Eq (3). In section 4, we prove the global convergence of the proposed method. In section 5, we report numerical results and present some comparisons for the existing methods to solve problem Eq (1). We conclude our paper in Section 6.

Throughout this paper, unless otherwise specified, ‖ ⋅ ‖ denotes the Euclidean norm of vectors or matrices.

Initial results

In this section, we first state some basic results in convex analysis and nonsmooth analysis. Let and denote p(x): = argmin_{z ∈ ℝⁿ} θ(z, x). Then, p(x) is well defined and unique, as θ(z, x) is strongly convex. By Eq (2), F can be rewritten as

In the following, we denote g(x) = ∇F(x). Some important properties of F are given as follows:

1. F is finite-valued, convex and everywhere differentiable with (4)
2. The gradient mapping g : ℝⁿ → ℝ is globally Lipschitz continuous with modulus λ, i.e., (5)
3. x solves Eq (1) if and only if ∇F(x) = 0, namely, p(x) = x.

It is obvious that F(x) and g(x) can be obtained through the optimal solution of argmin_{z ∈ ℝⁿ} θ(z, x). However, the minimizer of θ(z, x), p(x) is difficult or even impossible to solve for exactly. Thus, we cannot compute the exact value of p(x) to define F(x) and g(x). Fortunately, for each x ∈ ℝⁿ and any ϵ > 0, there exists a vector p^α(x, ϵ) ∈ ℝⁿ such that (6) Thus, we can use p^α(x, ϵ) to define respective approximations of F(x) and g(x) as follows, when ϵ is small, (7) and (8) The papers [36, 37] describe some algorithms to calculate p^α(x, ϵ). The following remarkable feature of F^α(x, ϵ) and g^α(x, ϵ) is obtained from [38].

Proposition 2.1 Let p^α(x, ϵ) be a vector satisfying Eq (6), and F^α(x, ϵ) and g^α(x, ϵ) are defined by Eqs (7) and (8), respectively. Then, we obtain (9) (10) and (11)

The relations Eqs (9), (10) and (11) imply that F^α(x, ϵ) and g^α(x, ϵ) may be made arbitrarily close to F(x) and g(x), respectively, by choosing the parameter ϵ to be small enough.

Second, recall that when f is smooth, the quasi-Newton secant method is used to solve problem Eq (1). The iterate x_k satisfies ∇f_k + B_k(x_k+1 − x_k) = 0, where ∇f_k = ∇f(x_k), B_k is an approximation Hessian of f at x_k, and the sequence of matrix {B_k} satisfies the secant equation as follows. (12) where y_k = ∇f_k+1− ∇f_k and s_k = x_k+1 − x_k. However, the function values are not exploited in Eq (12), which the method solves by only using the gradient information. Motivated by the above observations, we hope to develop a method that uses both the gradient information and function information. This problem has been studied by several authors. In particular, Wei, Li and Qi [39] proposed an important modified secant equation by using not only the gradient values but also the function values, and the modified secant is defined as (13) where ν_k = y_k + β_k s_k, f_k = f(x_k), ∇f_k = ∇f(x_k), and . When f is twice continuously differentiable and B_k+1 is updated by the BFGS formula [40–43], where B_k = I is a unit matrix if k = 0, this secant Eq (13) possesses the following remarkable property: This property holds for all k. Based on the result of Theorem 2.1 [39], Eq (13) has an advantage over Eq (12) in this approximate relation.

The new model

In this section, we present a modified BFGS formula using trust region model for solving Eq (1), which is motivated by the Moreau-Yosida regularization (smoothing), general trust region method and the new secant Eq (13). First, we describe the trust region method. In each iteration, a trial step d_k is generated by solving an adaptive trust region subproblem, in which the values of the gradient of F(x) at x_k and Eq (13) are used: (14) where the scalar ϵ_k > 0 and Δ_k describe the trust region radius.

Let d_k be the optimal solution of Eq (14). The actual reduction is defined by (15) and we define the predict reduction as (16) Then, we define r_k to be the ratio between Are d_k and Pre d_k (17)

Based on the new secant Eq (13) and with B_k+1 being updated by the BFGS formula, we propose a modified BFGS formula. The B_k+1 is defined by (18) where s_k = x_k+1 − x_k, y_k = g^α(x_k+1, ϵ_k+1) − g^α(x_k, ϵ_k), ν_k = y_k + β_k s_k and if k = 0, then B_k = I, and I is a unit matrix.

We now list the steps of the modified trust region algorithm as follows.

Algorithm 1.

Step 0. Choose x₀ ∈ ℝⁿ, 0 < σ₁ < σ₂ < 1, 0 < η₁ < 1 < η₂, λ > 0, 0 ≤ ɛ ≪ 1, Δ_max ≥ Δ₀ > 0 is called the maximum value of trust region radius, B₀ = I, and I is the unit matrix. Let k: = 0.

Step 1. Choose a scalar ϵ_k+1 satisfying 0 < ϵ_k+1 < ϵ_k, and calculate p^α(x_k, ϵ_k), . If x_k satisfies the termination criterion ‖g^α(x_k, ϵ_k)‖ ≤ ɛ, then stop. Otherwise, go to Step 2.

Step 2. d_k solves the trust region subproblem Eq (14).

Step 3. Compute Are d_k, Pre d_k, r_k using Eqs (15), (16) and (17).

Step 4. Regulate the trust region radius. Let

Step 5. If the condition r_k ≥ σ₁ holds, then let x_{k + 1} = x_k + d_k, update B_{k + 1} by Eq (18), and let k: = k + 1; go back to Step 1. Otherwise, let x_k+1: = x_k and k: = k + 1; return to Step 2.

Similar to Dennis and Moré [44] or Yuan and Sun [45], we have the following result.

Lemma 1 If and only if the condition holds, B_k+1 will inherit the positive property of B_k.

Proof “ ⇒ ” If B_k+1 is symmetric and positive definite, then “⇐” For the proof of the converse, suppose that and B_k is symmetric and positive definite for all k ≥ 0. We shall prove that x^T B_k+1 x > 0 holds for arbitrary x ≠ 0 and x ∈ ℝⁿ by induction. It is easy to see that B₀ = I is symmetric and positive definite. Thus, we have (19) Because B_k is symmetric and positive definite for all k ≥ 0, there exists a symmetric and positive definite matrix such that . Thus, by using the Cauchy-Schwartz inequality, we obtain (20) It is not difficult to prove that the above inequality holds true if and only if there exists a real number γ_k ≠ 0 such that , namely, x = γ_k s_k.

Hence, if Eq (20) strictly holds (and note that ), then from Eq (19), we have Otherwise, ; then, there exists γ_k such that x = γ_k s_k. Thus, Therefore, for each 0 ≠ x ∈ ℝⁿ, we have x^T B_k+1 x > 0. This completes the proof.

Lemma 1 states that if , then the matrix sequence {B_k} is symmetric and positive definite, which is updated by the BFGS formula of Eq (18).

Convergence analysis

In this section, the global convergence of Algorithm 1 is established under the assumption that the following conditions are required.

Assumption A.

1. Let the level set Ω
2. F is bounded from below.
3. The matrix sequence {B_k} is bounded on Ω, which means that there exists a positive constant M such that
4. The sequence {ϵ_k} converges to zero.

Now, we present the following lemma.

Lemma 2 If d_k is the solution of Eq (14), then (21) Proof Similar to the proof of Lemma 7(6.2) in Ma [46]. Note that the matrix sequence {B_k} is symmetric and positive definite; then, we present to be a Cauchy point at iteration point x_k, which is defined by where . It is easy to verify that the Cauchy point is a feasible point, i.e., .

If , then and Thus, we obtain Otherwise, we have . Thus, we obtain

Let d_k be the solution of Eq (14). Because , we have This completes the proof.

Lemma 3 Let Assumption A hold true and the sequence {x_k} be generated by Algorithm 1. If d_k is the solution of Eq (14), then (22) Proof Let d_k be the solution of Eq (14). By using Taylor expansion, F^α(x_k + d_k, ϵ_k+1) can be expressed by (23) Note that with the definitions of Are d_k and Pre d_k and by using Eq (23), we have The proof is complete.

Lemma 4 Let Assumption A hold. Then, Algorithm 1 does not circle in the inner cycle infinitely.

Proof Suppose, by contradiction to the conclusion of the lemma, that Algorithm 1 cycles between Steps 2 and 5 infinitely at iteration point x_k, i.e., r_k < σ₁ and that there exists a scalar ρ > 0 such that ‖g^α(x_k, ϵ_k)‖ ≥ ρ. Thus, noting that 0 < η₁ < 1, we have By using the result Eq (22) of Lemma 3 and the definition of r_k, we obtain which means that we must have r_k ≥ σ₁; this contradicts the assumption that r_k < σ₁, and the proof is complete.

Based on the above lemmas, we can now demonstrate the global convergence of Algorithm 1 under suitable conditions.

Theorem 1 (Global Convergence). Suppose that Assumption A holds and that the sequence {x_k} is generated by Algorithm 1. Let d_k be the solution of Eq (14). Then, holds, and any accumulation point of x_k is an optimal solution of Eq (1).

Proof We first prove that (24) Suppose that g^α(x_k, ϵ_k) ≠ 0. Without loss of generality, by the definition of r_k, we have (25) Using Taylor expansion, we obtain When Δ_k > 0 and small enough, we have (26) Suppose that there exists ω₀ > 0 such that ‖g^α(x_k, ϵ_k)‖ ≥ ω₀. By contradiction, using Eqs (25) and (26) and Lemma 2, we have (27) which means that there exists sufficiently small such that for each k, and we have ∣r_k − 1∣ < 1 − σ₂, i.e., r_k > σ₂. Then, according to the Algorithm 1, we have Δ_k+1 ≥ Δ_k.

Thus, there exists a positive integer k₀ and a constant ρ₀ for arbitrary k ≥ k₀ and satisfying , for which we have (28)

On the other hand, because F is bounded from below, and supposing that there exists an infinite number k such that r_k > σ₁, by the definition of r_k and Lemma 2, for each k ≥ k₀, which means that Δ_k → 0 for k → ∞; this is a contradiction to Eq (28).

Moreover, suppose that for sufficiently large k, we have r_k < σ₁. Then, , and we can see that Δ_k → 0 for k → ∞; this is also a contradiction to Eq (28). The contradiction shows that Eq (24) holds.

We now show that holds. By using Eq (11), we have Together with Assumption A(iv), this implies that (29)

Finally, we make a final assertion. Let x* be an accumulation point of {x_k}. Then, without loss of generality, there exists a subsequence {x_k}_K satisfying (30) From the properties of F, we have Thus, by using Eqs (29) and (30), we have x* = p(x*). Therefore, x* is an optimal solution of Eq (1). The proof is complete.

Similar to Theorem 3.7 in [25], we can show that the rate of convergence of Algorithm 1 is Q-superlinear. We omit this proof here (the proof of the Q-superlinear convergence can be found in [25]).

Theorem 2 (Q-superlinear Convergence) [25] Suppose that Assumption A(ii) holds, that the sequence {x_k} is generated by Algorithm 1, which has a limit point x*, and that g is BD-regular and semismooth at x*. Furthermore, suppose that ϵ_k = o(‖g(x_k)‖²). Then,

1. x* is the unique solution of Eq (1);
2. the entire sequence {x_k} converges to x* Q-superlinearly, i.e.,

Results

In this section, we test our modified BFGS formula using a trust region model for solving nonsmooth problems. The type of nonsmooth problems addressed in Table 1 can be found in [47–53]. The problem dimensions and optimum function values are listed in Table 1, where “No.” is the number of the test problem, “Dim” is the dimension of the test problem, “Problem” is the name of the test problem, “x₀” is the initial point, and “f_ops(x)” is the optimization function evaluation. Here, the modified algorithm was implemented using MATLAB 7.0.4, and all numerical experiments were run on a PC with CPU Intel CORE(TM) 2 Duo T6600 2.20 GHZ, with 2.00 GB of RAM and with the Windows 7 operating system.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Problem descriptions for test problems.

https://doi.org/10.1371/journal.pone.0140606.t001

To test the performance of the given algorithm for the problems listed in Table 1, we compared our method with the trust region concept (BT) of paper [15], the proximal bundle method (PBL) of paper [17] and the gradient trust region algorithm with limited memory BFGS update (LGTR) described in [26]. The parameters were chosen as follows: σ₁ = 0.45, σ₂ = 0.75, η₁ = 0.5, η₂ = 4, λ = 1, Δ₀ = 0.5 < Δ_max = 100 and (where k is the iterate number). We stopped the algorithm when the condition ‖g^α(x, ϵ)‖ ≤ 10 ^{− 6} was satisfied. Based on the idea of [26], we use the function fminsearch in MATLAB for solving min θ(z, x). Then, we obtained the solution p(x); moreover, we obtained g^α(x, ϵ), which is computed using Eq (8). Meanwhile, we also listed the results of PBL, LGTR, BT and our modified algorithm in Table 2. The numerical results of PBL and BT can be found in [17], and the numerical results of LGTR can be found in [26]. The following notations are used in Table 2: “NI” is the number of iterations; “NF” is the number of the function evaluations; “f(x)” is the function value at final iteration; “——” indicates that the algorithm fails to solve the problem; and “Total” denotes the sum of the NI/NF.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Test results.

https://doi.org/10.1371/journal.pone.0140606.t002

The numerical results show that the performance of our algorithm is superior to those of the methods in Table 2. It can be seen clearly that the sum of our algorithm relative to NI and NF is less than the other three algorithms. The paper [54] provides a new tool for analyzing the efficiency of these four algorithms. Figs 1 and 2 show the performances of these four methods relative to NI and NF of Table 2, respectively. These two figures prove that Algorithm 1 provides a good performance for all the problems tested compared to PBL, LGTR and BT. In sum, the preliminary numerical results indicate that the modified method is efficient for solving nonsmooth convex minimizations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Performance profiles of these methods (NI).

https://doi.org/10.1371/journal.pone.0140606.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Performance profiles of these methods (NF).

https://doi.org/10.1371/journal.pone.0140606.g002

Conclusion

The trust region method is one of the most efficient optimization methods. In this paper, by using the Moreau-Yosida regularization (smoothing) and a new secant equation with the BFGS formula, we present a modified BFGS formula using a trust region model for solving nonsmooth convex minimizations. Our algorithm does not compute the Hessian of the objective function at every iteration, which decrease the computational workload and time, and it uses the function information and the gradient information. Under suitable conditions, global convergence is established, and we show that the rate of convergence of our algorithm is Q-superlinear. Numerical results show that this algorithm is efficient. We believe that this algorithm can be used in future applications to solve non smooth convex minimizations.

Acknowledgments

This work is supported by China NSF (Grant No. 11261006 and 11161003), the Guangxi Science Fund for Distinguished Young Scholars (No. 2015GXNSFGA139001), NSFC No. 61232016, NSFC No. U1405254, and PAPD issue of Jiangsu advantages discipline. The authors wish to thank the editor and the referees for their useful suggestions and comments which greatly improve this paper.