Automatic differentiation features

Adept enables C++ algorithms to be automatically differentiated. For each mathematical statement involving scalars or arrays of a special "active" type, Adept stores the corresponding differential statement symbolically on a stack. The stack may then be used to perform the following computations:

• Full Jacobian matrix. Given the non-linear function y=f(x) coded in C or C++, Adept will compute the matrix H=∂y/x, where the element at row i and column j of H is H_i,j=∂y_i/∂x_j. This matrix will be computed much more rapidly and accurately than if you simply recompute the function multiple times perturbing each element of x one by one. The Jacobian matrix is used in the Gauss-Newton and Levenberg-Marquardt minimization algorithms.
• Reverse-mode differentiation. This is a key component in optimization problems where a non-linear function needs to be minimized but the state vector x is too large for it to make sense to compute the full Jacobian matrix. Atmospheric data assimilation is the canonical example in Meteorology. Given a non-linear function y=f(x) and a vector of adjoints ∂J/∂y (where J is the scalar function to be minimized), Adept will compute the vector of adjoints ∂J/∂x. Formally, ∂J/∂x is given by the matrix-vector product H^T∂J/∂y, but it is computed here without computing the full Jacobian matrix H. The adjoint may then be used in a quasi-Newton minimization scheme.
• Forward-mode differentiation. Given the non-linear function y=f(x) and a vector of perturbations δx, Adept will compute the corresponding vector δy arising from a linearization of the function f. Formally, δy is given by the matrix-vector product Hδx, but it is computed here without computing the full Jacobian matrix H. Note that Adept is optimized for the reverse case, so might not be as fast (and will certainly not be as economical in memory) in the forward mode as libraries written especially for that purpose.

Speed

The way that expression templates have been used and several other important optimizations mean that reverse-mode differentiation is significantly faster than other tools that provide equivalent functionality (ADOL-C, CppAD and Sacado) and less memory is used. In fact, Adept is also often only around 10-25% slower than an adjoint code you might write by hand, but immeasurably faster in terms of user time; adjoint coding is very time consuming and error-prone. For further details of the comparison with other tools, read the Adept paper.

Missing features

At present Adept is missing some functionality that you may require:

• Differentiation is first-order only: it cannot directly compute higher-order derivatives such as the Hessian matrix. However, in a large class of minimization problems the Hessian matrix can be computed from the Jacobian matrix (see chapter 4 in the User Guide).
• No capability to differentiate expressions involving complex numbers.

See also

• ADOL-C package
• CppAD package
• Sacado package
• Multiscatter package used in the Adept paper to evaluate Adept against other automatic differentation libraries
• autodiff.org: Community Portal for Automatic Differentiation
• Wikipedia page on Automatic Differentiation
• GNU Scientific Library, which includes an implementation of the fast L-BFGS quasi-Newton minimization method.