The André-Oort conjecture says that any subvariety of a Shimura variety with a Zariski dense set of CM points must itself be a Shimura subvariety. In recent years, this has been the subject of much work. We explain how this conjecture for the moduli space of principally polarized abelian varieties of some dimension $g$ follows from current knowledge and a conjecture of Colmez regarding the Faltings heights of CM abelian varieties--in fact its enough to assume an averaged version of the Colmez conjecture.