Models of neural networks can be largely divided into two camps. On one end, mechanistic models such as balanced spiking networks resemble activity regimes observed in data, but are often limited to simple computations. On the other end, functional models like trained deep networks can perform a multitude of computations, but are far removed from experimental physiology. Here, I will introduce a new framework for excitatory-inhibitory spiking networks which retains key properties of both mechanistic and functional models. The principal insight is to cast the problem of spiking dynamics in the low-dimensional space of population modes rather than in the original neural space. Neural thresholds then become convex boundaries in the population space, and population dynamics is either attracted (I population) or repelled (E population) by these boundaries. The combination of E and I populations results in balanced, inhibition-stabilized networks which are capable of universal function approximation. I will illustrate these insights with simple, geometric toy models, and I will argue that need to reconsider the very basics of how we think about neural networks.