Well....let me tell you from the beginning: This movie is probably on my top 5 list of the worst movies ever made.
First of all: I never really liked Milla Jovovich as an actress. She is always overacting like crazy - and I just think this movie is so overrated.
Secondly: Why is it that so many people I have talked to find this movie scary? It has 2, maybe 3 remotely scary scenes - one including an owl !!!!! - but apart from that it´s boring like crazy.
What I find even more annoying is the fact that from the beginning of the movie, the moviemakers are lying to the audience about the movie being authentical. But it isn´t.
We probably have quite different tastes in movies but I will offer my take and opinion on why this movie is (for me) both an interesting and a very frightening story.
Regarding Milla Jovovich, I agree she isn't the best actress. In fact, I never really cared about her being in the movie. Replace her with any other actress and I would still find it scary (hell, it might've even been a better movie that way).
Regarding the fakeness of the film, when I watched this movie the first time, I had no idea it was a fake. I was pretty sure it didn't really feature the material they claimed it was BUT I actually believed it could be maybe 1/4 or 1/3 true. Not flat out aliens but, you know, mysterious disappearances and whatnot.
Now, to the heart of the question, why do I (and maybe other people) find this movie scary? It's mainly because the aliens in it are shrouded in mystery. They are complete unknowns but are (supposedly) so terrifying, you'd go insane if you saw them and remembered that you did. The mystery combined with the dread they are supposed to evoke is what got to me in this movie.
Another (and i think the best) example is the owl. The abductees didn't remember being abducted but the owl evoked unspeakable terror in them anyway. That, in particular, evoked the worst kind of dread in me. The fact that anyone could be abducted and they wouldn't remember. Also, the tension is constant and the dread builds. It doesn't rely on sudden jumpscares and it's not as predictable as many ghost/serial killer horrors.
That is my opinion anyway. I can understand why many didn't find it scary (no judgement). It's just different wiring I guess. Some find the in-your-face daemon trying to kill you to be boring and find the unknown terrifying while other people are the opposite.