Neural field models can describe spatially organized activity in large populations of neurons. These models are integrodifferential equations where the kernel of the integral term describes the strength of synaptic connections between neurons. Traveling waves (Pinto and Ermentrout 2001), stationary pulses (Amari 1977), and spiral waves (Laing 2005) have all been identified as solutions to various neural field models. These analyses often employ two assumptions -- (1) effects of noise are small enough to be ignored and (2) connections between neurons only depend upon the distance between them (spatially homogeneous). However, recent studies have shown noise can cause traveling front (Bressloff and Webber 2012) and stationary bump (Kilpatrick and Ermentrout 2013) solutions to wander purely diffusively about their mean position. These analyses employ a small noise expansion technique developed for fronts evolving in stochastic PDEs (Armero et al 1998). The pure diffusion of spatially structured solution relies on the translation symmetry of the system, which only occurs in neural fields if synaptic connections are spatially homogeneous. We show that a variety of rich behaviors arise when one considers a wider variety of spatially structured synaptic connections. Namely, spatial heterogeneity establishes a multistable potential landscape in space that can then be traversed by solutions with the aid of noise. The dynamics then evolves as a particle kicked between multiple wells. In addition, considering long-range connections between multi-layered networks. Each layer supports a stationary bump solution due to its recurrent architecture. Connections between layers reenforce the initial position of those bumps in the presence of noise. Perturbative analysis of the effect of noise on the bumps allows one to derive a multidimensional mean reverting stochastic process for the position of the bump in each layer. Analytic and numerical results are discussed.