Wet hair is weaker and softer than dry hair and needs to be handled with care. Unfortunately towels aren't the most gentle tool you could use. The materials most towels are made of create grooves in the towel. Those grooves become aggressors to the hair cuticle and can pull out strands or cause split ends. Also towels absorb all of the moisture from the hair, leaving the hair frizzy.
Related post: Preventing the 4 Common Causes of Hair Damage
T-shirts on the other hand don’t have the aggressive grooves like towels. Instead they have a smoother, flatter surface. They also absorb excess water so they aren't removing all of the moisture from your hair.