Add proximal ADMM solvers (#371)

* Add reference * Work in progress on NonLinearPADMM * Docstring fixes * Docstring fixes * Add flexible partial function creation function * Add Function class * Various fixes * Typo fix * Add function supporting jvp with Hermitian transpose * Add algorithm iterations * Bug fix * Initial version of example script * Docs improvements * Bug fix * Typo fix * Fix usage example * Improve variable names * Docstring improvement * Add more accurate dual residual calculation * Add tests * Improve docs * Clean up unused module * Extend Function __init__ * Typing fix * Add slice method and tests * Improve jvp / vjp methods and add corresponding tests * Docstring fix * Add tests for complex values * Split out some tests into a new test module * Add function for constructing a jacobian linop for an operator * Add jacobian method and corresponding tests * Docstring fix * Change required by removal of Operator method * Rename conjugate vjp function * Fix docstring of cvjp * Minor docstring edit * Resolve typing issue * Resolve pytest issue * Update change summary * Change H type from Callable to Function and implement default values for x0, z0, u0 * Add jvp and vjp methods * Switch to using Function.vjp * Constraint function is now a Function object and explicit x0, z0, u0 initialization no longer required * Avoid example script title collision * Update example index * Change H type from Callable to Function * Add tests * Add handling of zero operators * Add parameter estimation utility function * Add tests * Update submodule * Add method constructing operator from function * Move parameter estimation functions to static methods of solvers * Rename module * Add proximal ADMM with linear constraint * Typo fix * Use linear-constraint version for baseline solver * Fix typo and add reference * Update comment on plot results * Update examples index * Update submodule * Fix or ignore typing errors * Update submodule * Docstring whitespace fix * Update submodule * Add change summary entry * Address test failures under osx * Improve test coverage * Address some review comments * Fix latex equation * Address PR review comments * Update submodule * Add discussion of effects of jit on timing * Update submodule Co-authored-by: Brendt Wohlberg <[email protected]>
lanl · Dec 13, 2022 · d05ff9c · d05ff9c
1 parent 2e64c7d
commit d05ff9c
Show file tree

Hide file tree

Showing 18 changed files with 1,815 additions and 46 deletions.
diff --git a/CHANGES.rst b/CHANGES.rst
@@ -6,8 +6,11 @@ SCICO Release Notes
 Version 0.0.4   (unreleased)
 ----------------------------
 
-• New methods and a function for computing Jacobian-vector products for
-  `Operator` objects.
+• New `Function` class for representing array-to-array mappings with more than
+  one input.
+• New methods and a function for computing Jacobian-vector products for `Operator`
+  objects.
+• New proximal ADMM solvers.
 
 
 

diff --git a/data b/data
diff --git a/docs/source/examples.rst b/docs/source/examples.rst
@@ -72,6 +72,7 @@ Miscellaneous
    examples/denoise_tv_admm
    examples/denoise_tv_pgm
    examples/denoise_tv_multi
+   examples/denoise_cplx_tv_nlpadmm
    examples/denoise_cplx_tv_pdhg
    examples/denoise_dncnn_universal
    examples/video_rpca_admm
@@ -120,6 +121,7 @@ Total Variation
    examples/denoise_tv_admm
    examples/denoise_tv_pgm
    examples/denoise_tv_multi
+   examples/denoise_cplx_tv_nlpadmm
    examples/denoise_cplx_tv_pdhg
 
 
@@ -182,6 +184,25 @@ Linearized ADMM
    examples/denoise_tv_multi
 
 
+Proximal ADMM
+^^^^^^^^^^^^^
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/denoise_tv_multi
+   examples/denoise_cplx_tv_nlpadmm
+
+
+Non-linear Proximal ADMM
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. toctree::
+   :maxdepth: 1
+
+   examples/denoise_cplx_tv_nlpadmm
+
+
 PDHG
 ^^^^
 

diff --git a/docs/source/references.bib b/docs/source/references.bib
@@ -83,8 +83,7 @@ @InCollection{beck-2010-gradient
   publisher =	 {Cambridge University Press},
   year =	 2010,
   doi =		 {10.1017/CBO9780511804458.003},
-  url =
-                  {http://www.math.tau.ac.il/~teboulle/papers/gradient_chapter.pdf}
+  url =          {http://www.math.tau.ac.il/~teboulle/papers/gradient_chapter.pdf}
 }
 
 @Book {beck-2017-first,
@@ -110,6 +109,17 @@ @Software {bradbury-2018-jax
   year =	 {2018}
 }
 
+@InProceedings {benning-2016-preconditioned,
+  title =	 {Preconditioned {ADMM} with nonlinear operator
+                  constraint},
+  author =	 {Benning, Martin and Knoll, Florian and
+                  Sch{\"o}nlieb, Carola-Bibiane and Valkonen, Tuomo},
+  booktitle =	 {IFIP Conference on System Modeling and Optimization (CSMO) 2015},
+  pages =	 {117--126},
+  year =	 2016,
+  doi =		 {10.1007/978-3-319-55795-3_10}
+}
+
 @Article {boyd-2010-distributed,
   title =	 {Distributed optimization and statistical learning
                   via the alternating direction method of multipliers},
@@ -206,6 +216,20 @@ @Article {daubechies-2004-iterative
   doi =		 {10.1002/cpa.20042}
 }
 
+@Article {deng-2015-global,
+  author =	 {Wei Deng and Wotao Yin},
+  title =	 {On the Global and Linear Convergence of the
+                  Generalized Alternating Direction Method of
+                  Multipliers},
+  journal =	 {Journal of Scientific Computing},
+  year =	 2015,
+  month =	 May,
+  volume =	 66,
+  number =	 3,
+  pages =	 {889--916},
+  doi =		 {10.1007/s10915-015-0048-x},
+}
+
 @Article {esser-2010-general,
   author =	 {Ernie Esser and Xiaoqun Zhang and Tony F. Chan},
   title =	 {A General Framework for a Class of First Order
@@ -345,10 +369,10 @@ @Article {kamilov-2022-plug
                   T. Buzzard and Brendt Wohlberg},
   title =	 {Plug-and-Play Methods for Integrating Physical and
                   Learned Models in Computational Imaging},
-  journal =	 {IEEE Signal Processing Magazine},
+  journal =      {IEEE Signal Processing Magazine},
   year =	 2022,
   eprint =	 {arXiv:2203.17061},
-  note =	 {To appear.}
+  note =         {To appear.}
 }
 
 @Article {liu-2018-first,
@@ -375,7 +399,7 @@ @Article {maggioni-2012-nonlocal
   number =	 1,
   pages =	 {119--133},
   year =	 2012,
-  doi =		 {10.1109/TIP.2012.2210725}
+  doi =          {10.1109/TIP.2012.2210725}
 }
 
 @InProceedings {makinen-2019-exact,
@@ -428,7 +452,7 @@ @Book {nocedal-2006-numerical
 
 @Book {paganin-2006-coherent,
   doi =		 {10.1093/acprof:oso/9780198567288.001.0001},
-  isbn =	 9780198567288,
+  isbn =	 {9780198567288},
   year =	 2006,
   month =	 Jan,
   publisher =	 {Oxford University Press},
@@ -541,7 +565,7 @@ @Article {valkonen-2014-primal
   journal =	 {Inverse Problems},
   volume =	 30,
   number =	 5,
-  pages =	 055012,
+  pages =	 {055012},
   year =	 2014,
   doi =		 {10.1088/0266-5611/30/5/055012}
 }

diff --git a/examples/scripts/README.rst b/examples/scripts/README.rst
@@ -76,8 +76,10 @@ Miscellaneous
       Total Variation Denoising with Constraint (APGM)
    `denoise_tv_multi.py <denoise_tv_multi.py>`_
       Comparison of Optimization Algorithms for Total Variation Denoising
+   `denoise_cplx_tv_nlpadmm.py <denoise_cplx_tv_nlpadmm.py>`_
+      Complex Total Variation Denoising with NLPADMM Solver
    `denoise_cplx_tv_pdhg.py <denoise_cplx_tv_pdhg.py>`_
-      Complex Total Variation Denoising
+      Complex Total Variation Denoising with PDHG Solver
    `denoise_dncnn_universal.py <denoise_dncnn_universal.py>`_
       Comparison of DnCNN Variants for Image Denoising
    `video_rpca_admm.py <video_rpca_admm.py>`_
@@ -140,8 +142,10 @@ Total Variation
       Total Variation Denoising with Constraint (APGM)
    `denoise_tv_multi.py <denoise_tv_multi.py>`_
       Comparison of Optimization Algorithms for Total Variation Denoising
+   `denoise_cplx_tv_nlpadmm.py <denoise_cplx_tv_nlpadmm.py>`_
+      Complex Total Variation Denoising with NLPADMM Solver
    `denoise_cplx_tv_pdhg.py <denoise_cplx_tv_pdhg.py>`_
-      Complex Total Variation Denoising
+      Complex Total Variation Denoising with PDHG Solver
 
 
 Sparsity