Value function methods have proven effective for learning high-dimensional optimal feedback controllers, but they fundamentally rely on solving the Hamiltonian maximization problem in closed form. Many practical systems, from space shuttle reentry to multi-agent swarm dynamics, involve Hamiltonians that admit no such explicit solution, limiting the applicability of existing methods. This talk presents a framework that overcomes this bottleneck by parameterizing the value function in an implicit neural network and training via Jacobian-Free Backpropagation (JFB). The approach preserves the structural advantages of value function parameterization: optimal controls are recovered using feedback form through the connection between Pontryagin's Maximum Principle and dynamic programming, enforcing physical consistency by construction. On the theoretical side, JFB is shown to converge to the stationary points of the expected optimal control objective. Empirically, we demonstrate that the framework is effective for high-dimensional problems, including multi-agent optimal consumption, swarm-based quadrotor and bicycle control.