The formula is

549

-54

495

Subtract two from three digits, and subtract one from the first digit at most. So a triangle is bigger than a circle. Look at the ten digits of three numbers, assuming that the digits don't borrow 1, you can guess that the square is 9. The square of a unit is equal to the sum of two other numbers, that is, a triangle is 5 and a circle is 4.