It should be possible. There are two branches of architecture, one is for design, and the other is for building structure (this part of course needs mathematics). Anyway, the high number in our school is C, which means less class hours and is relatively simple, but we can learn building mechanics and use mathematical knowledge.

Math is very useful. I suggest you study hard, whether it's design or postgraduate study.