It may be because the pavilion is a part of the building, just like the study now, and its area is smaller than other main rooms. If there are many books, it is called "library building". There are many Confucian classics to be stored in the temple, and the house to be stored must be very large, so it is called "library building". I think that's what it means.