I think Madison leaving in Season 4, along with them trying stylized storytelling. Instead of just telling the story, they got hung up on trying to find a different frame for each episode to make it "clever"
For me, it mostly failed from the beginning, then S4 came around and revitalized/reinvented/re-everything'd it into fucking brilliance. S5 managed to keep that up for a little bit, but it slowly dwindled again into blah. But man, S4 with Morgan and John and Al and the back-and-forth between their stories and the baseball stadium, that was an outstanding season of television.
They heard the piano and started falling off the balconies from the rooms. As though there had never been a noise before the piano that would have drawn them off the balcony. Sigh.
I thought the season after that where John Dorie and Morgan met up was the best season of that show. I was just trying to watch episode 2 of the new season, and it is truly horrible and revolting. They should just shut it down.