I know he was one of the villains, but Otto Kruger's character not only left his granddaughter completely unsupervised with the complete stranger he was in the process of framing, but left her stumbling around right next to a swimming pool!
You must remember that this was made in 1942, we can't use today's political correctness while watching a 1942 Hitchcock movie, plus its a spy-war movie not a lesson in parental responsibility. And in Hitchcock form it adds to the tension of the film, and the villan knew Robert Cummings would'nt do anything, he needed information.
Hitchcock needed the baby to get the letters and telegram out of the robe. So he had to have grandpa leave for a few seconds so that could happen and Kane would read them. And we see in the film that Kane does watch the baby all the time grandpa was gone.