Favreau has been riding Ironman's success for too long
Elf was okay but obviously driven primarily by Ferrell. Many directors could have handled that script similarly.
Ironman was good. It has a good script as well and again benefited hugely from the perfect casting of Robert Downey Jr in a role he was born to play.
But he's been downhill since. Matthew Vaughn has done more with less and seems more deserving of Star Wars. He's had lower budgets and worked with several young or lesser known actors and pulled terrific performances out of them.
Put simply, Vaughn could have directed Ironman and people would have done as good a job or better but if Favreau had done Layer Cake, Stardust, Kickass, or Xmen: First Class they wouldn't have been as good. Vaughn's just a better director overall and that's why he should have Star Wars. Favreau is just a puppet for the studio. Vaughn has vision.