This article describes the damage climbers do to plants when they climb rocks on which plants grow. I kinda had already guessed that rock faces used a lot for climbing were not full of plant life.
"Most climbers try to be responsible, but they are prepared to have a certain level of impact."
So as a climber, you do impact the rock. Just try not to impact the ground when you fall.