I don't believe for 1 second...

Arthur entering Sophie's apartment unannounced is part of the films reality; it set in motion the montage of prior moments/scenes that were not.

The latter insinuates that the former also happened in Arthur's mind, which would make sense given Sophie's reaction who, assuming that their encounter and/or conversation in the elevator was not among Arthur's delusions, had been friendly to Arthur in their first/original scene together. If anything, the elevator scene is the one that may be open to interpretation since it could be the beginning of Arthur's psychosis, not their final scene.

Furthermore, the scene is crucial to Arthur's character development because it emphasizes how far along his psychosis is. Arthur entered her apartment after he "had a really bad day," looking for the only person in his life who, literally, in his mind, would comfort him. Even the producer, director and writer of the film, Todd Phillips, established that Sophie and Arthur's final scene happened by referring to it as part of the film's reality, stating that he likes that it has audiences wondering whether Arthur killed her and confirming that Arthur did not, since Arthur's intention is to kill only those who wronged him, thus excluding Sophie.

As for Sophie leaving her apartment door unlocked in a dangerous neighborhood, that sort of thing actually happens more often than it should. Even people in rough neighborhoods not only leave the doors to their homes unlocked deliberately, but can be forgetful.

Especially in apartment buildings like hers where the main door is locked and those without keys have to be buzzed in by its tenants.

i leave my house unlocked at every moment of everyday.

I like you, found it odd that she left her door unlocked to have just anybody wonder in. Especially, as you say, she seemed to live in a rough neighbourhood.

Saying that, I think the final encounter was meant to be real. It emphasised that the 'relationship' that he'd had with her had been in his mind only. I think the first encounter in the elevator was real, which is why he'd been drawn to her. It was meant to demonstrate Arthur's failing mental health.

I think with the door unlocked thing, you just have to accept it as one of those things that was needed to move the plot forward.

he was in her apartment multiple times,
probably a key by then