Theory on a unique attribue of Mr. Tagomi

I was watching one of the episodes with his family in America and the idea hit me that perhaps he or a tiny group of people have the attribute of only existing in one world at a time.

Clearly many people have versions of themselves in at least several worlds and what they are in one doesn't effect what they do in another. Some have even developed completely different personalities and loyalties.

At first I thought Tagomi was simply visiting our world or another Allied Victory one briefly (as happened or at least is implied in the book), but once we see his family, we learn that he sort of belongs there! That blew my mind.

His wife is accusing him of abandonment and trying to divorce him. But in his world, she's dead, so it seems to me that there is a version of her there who died. Then it gets a little confusing to me, but also makes sense in a weird way.

Was his period of "abandonment" simply the time he spent in the Axis Victory timeline? If he had been consistently gone since 1946, though, she could probably have had him declared dead in absentia or the judge would have simply granted her the divorce whether he signed it or not.

So that makes me think that he has been making regular trips to see her and his son in which he promises to stay home this time, but always runs off again. Then that makes me wonder what a coherent story for the family in our world (or a similar one) would be.

It would probably be hard for a Japanese family to immigrate to America right after the war due to suspicion and the fact that while laws restricting non-white immigrants were mildly liberalized in the 40s and 50s (in comparison with the 20s) it would still be very hard to get in.

When I first saw his wife living in a house in America, I thought that in this timeline, she married an American military person and Tagomi would really freak out when he saw the wedding pictures.

So I speculated that maybe he was someone with special skills and they'd made an exception for him. But if this was the case, wouldn't the US Government see his constant disappearances as highly suspicious in light of the Cold War? Even if they didn't think he was working for the Russians, he wouldn't be working much for them, either, if he's hardly ever around.

Finally, I came up with the idea that in our timeline, he and his wife were immigrants some time before the war and met in an internment camp. Living in America for decades would explain his English skills in one timeline which are explained in the other by the fact that he would have to have become good at English to work with the occupation.