So I don't get how or why Red Skull hosts the Soul Stone competition

I was wondering the same thing myself.

It adds a level of mystery to the stones.

The implication is that they aren't just stones but something alive and if they sense something in you, they will control you, not the other way around.

I'm reminded on Hellraiser. The idea there is that people will hear about this puzzle box that does something. Normal people will not bother trying to find it, but insanely obsessive driven people will look for it. When they find it, it's very hard to open, so only the insanely driven with figure it out. When they do open it, they will eventually get transformed into a demon based on their own obsession.

The power behind the stones knew the Skull wanted to be part of it so badly it made him part of it forever.

I suspect the Soul Stone was previously in use (we know guys like Odin, the Celestials, and Malekith all messed with the other Stones so it's natural to assume the Soul Stone was being held by someone). When the Soul Stone was put back at whatever time, there was no guardian. When the 1940's rolled around and the Tesseract was awakened after so many centuries, it decided to shove the Nazi asshole to Vormir to be the new guardian in preparation for the next guy.

Keep in mind, the Stones were all being reawakened between the 40's and the 10's. The Aether had been locked away for millions of years but the Convergence opened the gateway to it. Going by James Gunn's statements, the Orb was underwater for centuries on Morag due to cyclical flooding on that planet and everyone had to wait for the water to recede and that happened around the same time. Thor mentioned in Age of Ultron that the Stones were all being rediscovered around the same time and that it likely had something to do with fate. I'm guessing making Red Skull a guardian for the Soul Stone was part of it.

Red Skull is there to guide people to the Soul Stone because the script needed him to be. That's about the only reason I can think of.

I thought that was weak too. My memory tells me that it was punishment to RS via being put in its vicinity, never being able to obtain it, and eventually seeing it go away. But truthfully, I'm not certain. It has a random fan-service flavor like much of Rogue One.

Still though ... love both of these movies.

Maybe who ever was guarding was no more and the stone and teleported red skull when he touched it.

And what happens if someone shows up alone with no one to throw over the cliff? I assume they have no way to get it?

Red Skull said he too sought the stones which makes me think he went looking for the Soul Stone after being projected into space.

It probably wasn't possible for Red Skull to lose someone he loved to get the stone. An idea I had was that he killed the original guardian which made him trade places with it.