For me, there are two issues that twine together to increase this hatred of the phone: 1) I'm an introvert and 2) I rely heavily on body language to communicate. If you put those together, it becomes nearly impossible to have a conversation over the phone that consists of anything other than small chitchat (which I'm no good at and dislike) or appointment making.
The introversion means that I'd be perfectly content to have the entire conversation in my head and never actually communicate anything to anyone. In fact, this is part of the problem I have with posting regularly to LJ - once I have the post composed in my head it's sufficient and I never actually have to type it out. Some days I wonder if I could spend the entire day without speaking to anyone. Fortunately, that's thwarted by carpooling with my husband or I might get out of the practice of talking altogether.
I've also learned to rely very heavily on body language because sometimes the little censor in my head takes the day off and the weirdest stuff comes out of my mouth. Sometimes he'll kick back in right after the words jump out, but other times the only guide I have to the fact that I've said something off-color or that came out in a completely unintended way is the body language of those around me. This doesn't happen that often, but the silence in phone conversations frequently make me wonder if I've done just this and mortally offended someone. I aways end up asking questions like "are you still there?" or "is that ok?" far more often on the phone than I need to in person. The feedback loop just isn't there and it adds to my discomfort and raises the odds that I'll say something stupid. It's a vicious cycle.
I guess all of this is just an elaborate explanation for why I prefer one method of communication over another. And an explanation for why I never call people just to chat.