Of course, you can take it with you if you don't disturb others. It's best to bring your baby's usual small toys (the ones that don't make noise) and his favorite drinking pot when you bring them. You can't eat in the library, so you can't bring cookies or anything.