Moss is distractingly unattractive...
It seemed more fitting in Mad Men because it was acknowledged. She was just an average looking even dumpy woman in contrast to a lot of the trophy wives. Although one might argue her attention from men was a little absurd, none of her love interests were idols.
But it seems like her role here to be this highly desired woman and it just doesn't work. I think Yvonne should have been the handmaid.