Because of the formidable challenge of observing the full-depth global ocean circulation in its spatial detail and the many time scales of oceanic motions, numerical simulations play an essential role in quantifying patterns of climate variability and change. For the same reason, predictive capabilities are confounded by the high-dimensional space of uncertain inputs required to perform such simulations (initial conditions, model parameters and external forcings). Inverse methods optimally extract and blend information from observations and models. They enable formally calibrated and initialized predictive models to optimally learn from sparse, heterogeneous data while satisfying fundamental equations of motion. A key enabling computational approach is the use of derivative information (adjoints and Hessians) for solving nonlinear least-squares optimization problems with quantified uncertainties. Derivative information enables property-conserving data assimilation for reconstruction, adjoint-based dynamical attribution, and the use of Hessian information for uncertainty quantification and observing system design. A close correspondence exists to neural network training via backpropagation to learn from big data. Other correspondences exist between layer-wise relevance propagation for explainable AI and the interpretation of the time-evolving dual state of physical models and their use as sensitivity kernels for dynamical attribution. After highlighting these correspondences, we discuss a path forward to merge these perspectives via the concept of universal differential programming encapsulated in general-purpose automatic differentiation, such as being developed in the Julia programming language.