Something to bear in mind is that your opponents lose line of sight to that side of the map, which can allow you to ambush, and also cap their portal quickly and without them noticing. As soon as you take down the first tower, the next thing on your mind should be harassing their gold mine/celerity flag towers. Once one of those is down you can backdoor without being troubled too much, and the game will really open up.
In fact you can just run minions in front of you if you're a general in order to soak up damage from the gold mine flags while you run through to cap, so getting rid of the first tower really is important if that's a valid option for you.
300 extra gold is also very handy at level 4, you can now buy a flag lock 