It is good to refuse naturally, but it is also good to make an appointment at a reasonable time.

Just say, tomorrow? I have plans for tomorrow. Why don't we take a rain check? How about next week?

Usually this kind of rejection is pleasant, because he knows that you have a reason and didn't refuse him on purpose, and you have asked him to go with you next time. Of course, he will be very happy.