Computation and learning for discrete-time mean-field games (MFGs) have recently gained significant interests. However, most existing works on algorithm analysis are restricted to contractive or monotone settings. In this talk, we introduce MF-VMO (Mean-Field Visitation Measure Optimization), the first algorithm framework for MFGs that goes beyond these settings. Our approach is based on a novel reformulation of finding mean-field Nash equilibria as a smooth optimization problem. We show that finding (-) Nash equilibria for MFGs is equivalent to (approximately) solving the reformulated optimization problem. This opens up the possibility of solving MFGs with the rich literature and tools of optimization. We discuss the implementation of optimization procedures for solving the reformulated problem and derive local and global convergence guarantees for some popular algorithms including gradient descent and stochastic gradient descent. Finally, we show the flexibility of the framework by discussing the extensions to solving multi-population MFGs, discounted MFGs and refined Nash equilibria for MFGs.