Up until late game he is pretty slow about it. He will chip away at your towers, but almost every other demigod can do it. The generals have their archers, the torch bearer can fireball the tower, rook can just drink the tower (yes, he is getting hit, but it ends up a net positive for him.) The counter is simply scaring him off or killing him. While he is just outside of tower range, he isn't a great distance from the tower. If torch bearer and someone else both scroll to the tower he is plinking at the same time you can snare him and probably kill him.
Running around, completely unable to be caught, is more a symptom of the many stacking speed items. You can stack the anklet, the boots of speed, the improved boots of speed (last expensive item in footwear shop), and late game the cloak to reach untouchable speeds. Any hero except rook can move around with impunity with stacked speed. In my opinion stacked speed absolutely needs to be removed and heroes shouldn't be balanced around it staying.