We study the learning of finite mixture modeling in a singular setting,where the number of model parameters is unknown. Simply overfittingwithout suitable regularization of the model can result in highly inefficientparameter estimation behavior.To achieve the optimal rate of parameter estimation, we propose ageneral framework for estimating the mixing measure arising infinite mixture models, which we term minimum $Phi$-distance estimators.  We establish a general theory for the general minimum $Phi$-distanceestimator, which involves obtaining sharp probability bounds on theestimation error for the mixing measure in terms of the suprema ofthe associated empirical processes for a suitably chosen function class$Phi$. The theory makes clear the dual roles played by the functionclass $Phi$, rich enough to induce necessary strong identifiabilityconditions and small enough to produce optimal rates for parameterestimation.Our framework not only includes many existing estimation methods as specialcases but also results in new ones. For instance, it includes the minimumKolmogorov-Smirnov distance estimator as a special case, but also extendsto the multivariate setting.  It includes the method of moments as well,while extending existing results for Gaussian mixtures to a larger familyof probability kernels. Moreover, the minimum $Phi$-distance estimationframework leads to new methods applicable to complex (e.g., non-Euclidean)observation domains.  In particular, we study a novel minimum distanceestimator based on the maximum mean discrepancy (MMD), a particular$Phi$-distance that arises in a different context (of learningwith reproducing kernel Hilbert spaces).  This work is joint with Yun Wei and Sayan Mukherjee.