Silent people can be obtained by drawing cards. If you are lucky, you will draw out some library passes.
You can get a taboo card or a glory card for free if you draw a certain number of times.
Then you add taboo cards to the deck and use them more in combat.
In this way, your taboo value will go up, and the corresponding brilliance value will drop.