In this talk, we will present the structure on various spectral analysis of the quantum Rabi model (QRM) and its asymmetric version (AQRM), which are widely studied fundamental model of light-matter interactions. Also, we will discuss their covering mathematical model eta-NCHO (eta-shifted Non-Commutative Harmonic Oscillator), that has rich arithmetic properties such as in a closed relation with modular forms elliptic curves and Eichler forms, etc. at least when eta=0. Targets of the material on physical models are the discrete (explicit) path integral representation of the heat kernel (and partition functions) of the QRM, analytic continuation of the spectral zeta functions, and the hidden symmetry for AQRM in relation with the eta-NCHO. We will also give several conjectures arising from those study, which are related to representation theory of infinite symmetric group, Feynman path integrals, Diophantine Geometry and certain Fuchsian differential equations including their confluence pictures.