I think people who are bothered/offended by quiet people are really insecure (especially in the workplace).
I seriously hate the fact that in a lot of workplaces you can't sit in a room full of people and just want to be silent without someone taking offence and thinking you're being rude and thinking you're better than them. I enjoy being quiet and introverted. I don't feel the need to constantly fill every second of silence with some pointless chatter. I think people who expect you to always want to talk to them have really fragile egos and simply can't handle the fact that not everyone wants to chit chat all the got damn time.
I don't mind listening. The issue for me is when people think that you HAVE to have an opinion and vocalize that opinion or else you're being rude and snobby. Unfortunately, introverts are frequently misunderstood people in the workplace.