Most people think math is more difficult. College mathematics is more professional, more general and more abstract. Most theories have little connection with reality and only serve some specific models. In a word, mathematics is the foundation of natural science, and many scientific research work cannot be carried out because of poor development. So mathematics is abstract and difficult to learn.

College physics is mainly the advanced knowledge of communication and quantum mechanics. Relativity is certainly much more difficult than the classical knowledge of mechanics, optics and electricity in high school, because many of them are abstract, and students can't do experiments by themselves according to the experimental conditions of the school, so it is definitely much more difficult for most students.

This is my experience, not copying the existing answers.