My wife and I are having the same problem, but worse. We usually only try and play with a couple friends (never more than 4 players total), and if just one of us plays it works fine. If we both try and connect, we might get it to work 1-2 games out of 10. I have disabled "Filter Multicast" and left my wife's PC at the default ports but changed mine to 6133 - 6153, then set up port forwarding on my WRT54G for each port range to the specific static IP address for each of us. No luck so far, in fact now it is much worse, we havent gotten a single game to work. I have TWC and a giant pipe, so bandwidth is not an issue. Since I bought this game for the multiplayer to play with friends and I cannot do that, i feel very cheated.
Stardock, please, let me know what I need to do to fix this.