This scene bothered me

I know that Joe was loyal, stood up to everyone, family, his superior at hospital. Went and found another position, and wanted Martha to come with them. Showed many fine qualities. But the scene near the beginning when he comes over after the party and roots through the refrigerator showed he had no consideration for their money problems. But, what bothered me the most was when he came to get Karen to go out. Karen told him she and Martha were leaving, and he said "I can't deal with two girls" and then when she said "Martha and I were going for a walk" he sat down and very smugly said "Fine, go!" Very dismissive of Martha's plans to go for a walk with Karen, telling her "I don't want you along." This made me never really like his character.