So f [f (2)] = the square of 2A +aB = the square of A+A "

It should be 2a squared+ab+b.

Solution: let f (x) = ax+b.

From the meaning of the question: f( 1)=a+b= 1.

f(2)=2a+b，

f[f(2)]=a(2a+b)+b=a(a+(a+b))+b=a(a+ 1)+b=a^2+(a+b)=a^2+ 1

Let y = f (x) = ax+b and x = (y-b)/a, so f-1(x) = (x-b)/a.

f- 1(4)=(4-b)/a

So a 2+1= 2 (4-b)/a.

The solution is: a=2, b=- 1.

So f(x)=2x- 1.