I talk on the phone while skating all the time. The key is to cover up the mouth hole from lots of air so the person can actually hear you. I'm generally out of breath though when I'm talking.
I did it once. I was talking to my wife with a wireless bluetooth headphone on, hit a rock, stopphed and did a nice front breakfall. She didn't even notice. That was also the last time.