The past tense must be used 20 days ago. Although both the main clause and the attributive clause use the simple past tense, it is located in the attributive clause, in other words, it is closer to being borrowed in the attributive clause, so it should be modified. If you want to modify returned, you can move it to the beginning of the sentence: 20 days ago, he returned the book borrowed from the library.