The dispersive shallow water equations (DSWE) are fluid models, applicable to coastal regions that include additional physics (such as dispersion) to the well-known shallow water equations. The DSWEs present several challenges for efficient time-stepping including mixed space and time derivatives, nonlinearities and higher order spatial derivatives. We present semi-implicit high-order time stepping strategies that avoid a fully implicit treatment of the nonlinear terms and simplify the treatment of the mixed space-time derivatives. The approach is based on extensions of a recent unconditional stability theory, where due to the structure of the equations, zero stability plays the role of unconditional stability.