The family non-sense is stupid filler
The show doesn't have a clear focus. Most people don't care about the family stuff, it's very lame and non-essential to the plot. We do not know their life nor do we care to know. We are not watching soap-drama. It's frustrating to see these family non-sense interrupting the real spy story.
People who are interested in the espionage topic don't give a damn about some random teenage rebellion problems, and people who watches the typical dramas would not find the spy plot appealing and probably get confused and think it's too complicated.
This lack of awareness of what the intended audience is and what would they want to watch is what contributed to its downfall.