About "not so steady" internet connection

I noticed this after joining many games. If you have a "not so steady"(?) internet connection, you will be kicked automatically (disconnect); however I still can see other players chatting within the chat showing me that I am connected, yet I am not. (which I believe means I disconnected for very very very short amount of time) I can't type, I don't see my name.

This situation happen almost always in a scenario that me and my brother on LAN playing in the same room with other people (usually when the 6th person join); as someone join, this problem occur.

It would not occur when we are in 2 different game lobbies.

It would not occur when there are less than 5 people in the lobby.

It wouldo not occur when there is already someone with 0ms in the room.

I'm using vista 64.

Brother using XP 32.

Edit: it does occur under 5 people, but not often.