We consider systems of interacting agents or particles, which are commonly used for modeling across the sciences. While these systems have very high-dimensional state spaces, the laws of interaction between the agents may be quite simple, for example they may depend only on pairwise interactions, and only on pairwise distance in each interaction. We consider the following inference problem for a system of interacting particles or agents: given only observed trajectories of the agents in the system, can we learn what the laws of interactions are? We would like to do this without assuming any particular form for the interaction laws, i.e. they might be “any” function of pairwise distances, or other variables, on Euclidean spaces, manifolds, or networks. We consider this problem in the case of a finite number of agents, with observations along an increasing number of paths. We cast this as an inverse problem, discuss when this problem is well-posed, construct estimators for the interaction kernels with provably good statistically and computational properties. We discuss the fundamental role of the geometry of the underlying space, in the cases of Euclidean space, manifolds, and networks, even in the case when the network is unknown. Finally, we consider extensions to second-order systems, more general interaction kernels, stochastic systems, and to the setting where the variables (e.g. pairwise distance) on which the interaction kernel depends are not known a priori. This is joint work with Q. Lang (Duke), F. Lu (JHU), S. Tang (UCSB), X. Wang (JHU) , M.Zhong (IIT).