How might one infer circuit properties from neurophysiological data. We address this challenge for the mouse visual system with a novel neural manifold obtained using unsupervised machine learning algorithms. Each point on our manifold is a neuron; nearby neurons respond similarly in time to similar parts of a stimulus ensemble. This ensemble includes drifting gratings and flows, i.e. patterns resembling what a mouse would “see” while running through fields. Our manifold differs from the standard practice in computational neuroscience, of embedding trials in neural coordinates. Importantly, for our manifolds topology matters: from spectral theory we infer that, if the circuit consists of separate components, the manifold is discontinuous (illustrated with retinal data). If there is significant overlap between circuits, the manifold is nearly-continuous (cortical data). To approach real circuits, local neighborhoods on the manifold are identified with actual circuit components. For the retinal data we show these components correspond to distinct ganglion cell types by their mosaic-like receptive field organization, while for cortical data, neighborhoods organize neurons by type (excitatory/inhibitory) and anatomical layer. The manifold topology for deep CNN's will also be developed. Joint research with Luciano Dyballa (Yale), Marija Rudzite (Duke), Michael Styrker (UCSF) and Greg Field (UCLA).