When Jimmy gets back to London and his mates, he is still buzzing about the fight and getting arrested and sharing the dock with the Ace Face, but his mates are all, like, "whatever". Why are his fellow-mods not equally excited about their recent experience?
They didn't share Jimmy's idolistic views of mod culture. To Jimmy, Ace was some kind of god and he had something nearing a religious experience, but to everyone else it was just another weekend of booze, pills and fighting. That is why the end events hit Jimmy so hard too, to see his idol fallen, not a god but a bellboy. His whole world fell apart and how he perceived things he found out was just a lie.
Yes that's true, in fact his friend Dave says something like "we had to get back to our jobs", in other words we had to get back to our working lives, a life which Jimmy has been desperate to escape from throughout the film. Even his new girlfriend had more or less forgotten him.