You're right, I just played another game, and that was the problem.
Upgrading to priests early is still effective, as long as you make sure to destroy towers that are in the lanes after you do it. This was the problem I was having is that they were just being used as free exp and gold for the other team and I wasn't doing anything with it. If you destroy the towers slowly but surely, the priest upgrade will pay off, especially if you buy it in conjunction with unit HP upgrades. If you just send priests in and don't do a thing about the towers then you really are shooting yourself in the foot and the other team will just build up levels, gold, and war rank, and eventually come back and knock you off.
It was a 3v3 win, and knocking out the towers in the lanes allowed us to put too much pressure on them and they could not recover.