The theory of machine learning has been stimulated, in recent years, by a series of empirical observations that challenged the standard knowledge inherited by the classical statistical theory. In the first part of the talk, I will review some results on simple mean-field models, that allowed statisticians and physicists to understand some of these unexpected behaviors, e.g., the double-descent phenomenon or the effectiveness of ensembling. The models rely on some simplifying assumptions, one of them related to some kind of "Gaussianity of the dataset". In the second part of the talk, I will present therefore two recent works in which we characterized regression and classification tasks on fat-tailed datasets. We showed how Gaussian universality can break down and how non-Gaussianity can affect the generalization performances, for example, the generalization rates, the existence of an MLE in a classification task, and the robustness of a Huber estimator.