How ridiculous.
Midgets and dwarfs marry and reproduce all the time, and not always with other midgets or dwarfs. So it's plausible, especially if he's the only guy around.