In this talk, I will discuss several recent works from our group studying interpolating high-dimensional linear models. On the bright side, we show that for sparse ground truths, minimum-norm interpolators (including max-margin classifiers) can achieve high-dimensional asymptotic consistency and fast rates for isotropic Gaussian covariates. However, we also prove some caveats of such interpolating solutions in the context of robustness that are also observed for neural network learning: when performing adversarial training, interpolation can hurt robust test accuracy as compared to regularized solutions. Further, in the low-sample regime, the adversarially robust max-margin solution surprisingly can achieve lower robust accuracy than the standard max-margin classifier.