Sainbayar Sukhbaatar - New York University (NYU) Deep networks are often designed to take advantage of a certain input structure. For example, convolutional networks assume a spatial structure, while recurrent networks work on sequences. Although such specialization is good for the performance, it also means we need different architecture for other types of inputs. In this talk, we present a simple deep network architecture for sets. The network processes each set element separately in its own stream, except the streams can share information through a broadcast communication channel. We apply this model to multi-agent tasks, demonstrating the ability of the agents to learn to communicate amongst themselves, yielding improved performance over non-communicative agents and baselines.