You are good at physics, which means you are smart. The reason why you are not good at math is that your math foundation is not good. Can you remember your formulas and theorems? If you make a mistake once, you will be afraid next time, and your self-confidence will get worse and worse. What you need to do is to find your own self-confidence and establish your own mathematical thinking. Start with the basics, be familiar with the basic theory, and take your time.