Mathematics is a big field, with many branches, and the frontier problems of different branches are different. For example, theoretical physics also uses mathematics, so in this field, the frontier problems are line theory, matrix of N-dimensional space and so on, but if other branches, such as number theory, will study Riemann conjecture. You can refer to Hilbert's list of problems, and many problems have not been solved so far.

The following picture can also be used for reference.