Umbridge and the minister weren't working with Voldemort?

That's actually one of the things I like about the HP universe - people aren't clearly divided into good and bad. Yes, Voldemort's supporters are all bad, to a greater or lesser degree, but the good guys aren't all good either. Even Dumbledore, Voldemort's mightiest and most dedicated opponent, has slave labor at his school and is prepared to sacrifice Harry to the cause, he's not a nice or ethical person. He really is out for "the greater good".

And Umbridge is just a random asshole, someone who lives for petty tyranny, and who found more scope to be an asshole under Voldemort than under Fudge. I doubt she gives a rat's ass about politics or the big picture, she just wants to boss people around and punish anyone who refuses to obey.

As for Fudge, I suspect that a lot of his opposition to Harry is due to his long and difficult relationship with Dumbledore, which is never discussed in detail in the books, just mentioned here and there. Dumbledore had been offered the Ministry of Magic jobs several times and turned it down, which means that he would have been a direct rival of Fudge's, and Fudge would have resented him and thought of him as a rival. But worse, Dumbledore had eschewed the political world, instead of going into politics he'd chosen to make Hogwarts into his personal, unassailable power base. Which it was, he employed OooP members and allies even if they were terrible teachers (Snape), he manipulated the minds of a generation, and made the all-important Harry Potter into his unquestioning supporter. Now, if you were a politician, who'd spent your career in the public eye, what would *you* think of someone who used the nation's educational system to build a secret power base??? No, Dumbeldore was *not* a nice man, and a decent and rational man like Fudge could easily mistrust him...