Why Does Joe Get Such A Bad Rap?

He was pretty damned loyal throughout the film -- defying his aunt, losing his job (which he could have easily re-gained by 'cutting ties'), even insisting -- INSISTING -- that the Shirley Maclaine character -- the cause of all the trouble -- come along with them to relocate after he sacrifices his promising medical career to start a new life as the local doc in some backwater town. Note that he sticks by them even after Rosalie 'corroborates' the 'truth' of the rumor. AND he shows up at the funeral, when he could have been long gone.

Then all of a sudden he's a bad guy for simply wondering whether they may have been some kernel of truth to it? Which he asks only after Karen begged him to be honest about it? In fact, the fact that Karen is so repulsed by the idea and the thought that he ever wondered about it says a hell of a lot more about her than him.