I'm seeing this problem as being more prevalent: I try joining a game, connect to about 5 people, but not to say a 6th person. So then we try reconnecting, and perhaps we don't connect to some other guy.
To me I'd first say the matchmaking is just screwy, why won't it even SEE the player (he just isn't even attempted to be connected to)
Even hosting, some players just don't even see me to connect to, while others do. These are players who I played a game last time with; I know they're fine.