Because this is the starting point.
Back means to go back and go to the original place.
So it's going back, "to" a new destination. For example, it will go to someone's home.
This should be understood literally.