You're too rough with the towel.
Intense towel drying can ruin your hair follicles, which can lead to damage and breakage. And, it can also worsen skin conditions like dryness and itchiness. Pat dry whenever possible and avoid wrapping your hair up in a towel.