I think it depends on the field of "agree with." I don't need to agree with my friends on everything, but there are ethical questions that could probably break a friendship. Also, there's a useful distinction between "agree with someone about book X" and "agree that books are worth talking about."