Phase retrieval has grown in popularity in recent years on account of its success in x-ray coherent diffractive imaging, including ptychography. Our development of phase retrieval for image reconstruction will be reviewed, including both passive and active imaging, in both optics and x-rays, enabling imaging from the nano-scale to the astronomical scale. Iterative transform and nonlinear optimization algorithms will be compared.