In DH2, John had transferred to LAPD and things were peachy with Holly. This film seems to ignore that. It's like he never left NYPD and never made up with Holly. When it was in theaters, some people said it took place between DH1 and DH2. WTF??
Some people have speculated that McTiernan consciously decided to ignore the second film. (he wasn't the director of Die Hard 2) Compounding that notion with the fact that the installment designation number of "3" was dropped from the title. I don't know if any of that is true, but it's worth noting. Even if it does not consciously acknowledge 2, I don't think it outright contradicts it either. Though they don't go into detail explaining the situation of him leaving the LAPD, I figure it can be inferred without much of a hitch that he and Holly were again having problems and thus he transferred back. It's almost open-ended enough to be as if one can choose whether or not to count 2 in this film's timeline.
I didn't see it as a problem. McClane specifically told Zeus that Holly was in LA, that they needed space and he came back to New York. He can't come back to New York if he wasn't away in the first place, so it implies that he was indeed in LA. He doesn't go into a lot of detail, but he doesn't have to. For people who've seen all the films, they know he was in LA, so the line works. For people who didn't see any of the other films, or even if they didn't see the second, it still works for the purposes of this story.
Apparently as weird as this sounds you don't have to take my word for it I could be wrong but Die Hard With A Vengeance is a direct sequel to Die Hard 2: Die Harder. I remember a while back watching Die Hard With a Vengeance and noticed John McClane has a scar on his left eyebrow which he sustained from the second film if I remember correctly, nice bit of continuity there! It's fairly noticeable during the payphone scene where Simon says, "You could have simply said there was a fat woman!" and the camera shows a close up of McClane's face you can see he has a scar on his eyebrow! I'm going to double check now actually just in case I'm wrong haha!