Is it really bad to vigorously towel-dry my hair?

I have medium-length hair and I hate using product, so the only way for me to get it to look the way I like (crazy/pouffy/bed-headish) is to towel-dry it pretty vigorously. I know that's not terribly good for it, but I don't really care if I get split ends or whatever. My only concern is that maybe there will be negative long-term effects, like hair loss perhaps. Does anyone have any insight on this?