We propose the use of image symmetries, in the sense of equivalences under image transformations, as priors for learning symmetry adapted representations, i.e., representations that are invariant to these transformations. We show how Deep Convolutional Neural Networks implement such representations for the translation group and propose a new regularization term to extend the learning to other groups. Further, from a computational neuroscience point of view, we show in which sense the ventral stream architecture can be mapped to a class of Deep Convolutional Neural Networks.