I think I know why he dislikes Jim Carrey
The rift between him and Carrey on the set of Batman Forever is well known. However, I read Jim walked into a restaurant and Tommy walked up to him, hugged him and told him "I cannot sanction your buffoonary."
Jim Carrey acts like a crazy person a lot of the time, so I think Tommy does not like crazy people.