You won't find a solution to it on your end, its purely Stardock at fault and it will require them to fix it. Just going to have to hold your hats and pray they fix it quickly.
That's not looking likely though, considering only a handful of people have this problem opposed to the huge percentage of the playerbase that can't even connect to a game. I'm usually the last person to point a finger at the game, but I think I've really gone above and beyond my user end in regard to trying to solve this problem. Absolutely nothing even dents it, even when most of the variables are gone.
This problem only occurs when more than three other users are connected in a game. 2v2 games are totally devoid of this problem I have found. In a 3v3 game, if two people drop, the spikes stop immediately. The more people, the more frequent the spikes.