Not necessarily. Mathematics is the foundation of science. If math is good, physics won't be too bad. Poor math will affect the difficulty of physics learning, but physics can still be learned well! I was poor at math, and I didn't know physics at first. I went to extra-curricular tutoring, and physics was much better than math in the end. I got good grades in chemistry, so there is no absolute connection.