(-A-2B)cosx+(2A-B)sinx=cosx
Therefore, -A-2B= 1, 2A-B=0, while A=- 1/5 and B=-2/5.
So y * =- 1/5e x (cosx+2sinx).
Method 2: e^xcosx is the real part of e (( 1+I) x), so first find the special solution of y'-4y'+3y = e (( 1+I) x) and set it as y * = AE ((65438)).
So y * = (-1/5+2i/5) e ((1+i) x) = (-1/5+2i/5) e x (cosx+isinx), which is actually -65438+.
So a special solution of y''-4y'+3y = e x cosx y * =-1/5ex cosx-2/5exsinx.