Personally, I don't think it is necessary to change majors. If you really feel that you are not interested in it, change it.
Mathematics department has more prospects for further study in the future!
To tell the truth, mathematics belongs to science after all. Of course, undergraduates are not as good as engineering undergraduates at the same level!
So further study is better!