That is the bandwidth graph from yesterday of JUST the multimedia/movies module.  It's measured in megabits. So we peeked at over 300 megabits per second yesterday on just that.  To put things in perspective, Galactic Civilizations I required a total of 30 megabits of bandwidth for the entire game. 

You can see where at around 4pm Cogent, our ISP for our database server, hic-upped causing the default server to not transmit.  At 4:15pm, you had numerous people posting on the forum about how bad their download experience was and when it came back at 4:30pm, everything was fine again.

Observations

I would make a terrible tech support person. I just don't have the patience for it.  I'm an engineer so I tend to get riled at hyperbole in descriptions.  So here are some things I've noticed:

#1. Not only will people not read the manual. They won't read the readme. They also won't read the sticky posts with titles such as "STICKY: Don't have sound? Make sure you download the multimedia!".  I took a screenshot last night where the forum list literally had two "NO SOUND!!11" type posts in a row that were directly underneath the sticky.

#2 Similar to #1, people won't update their video drivers no matter how many times you tell them.  One user, who posted several different places how "buggy" the game was because it kept crashing on them finally, after much loud complaining, gave us a debug.err.  He had an ATI card with drivers from 2003.  His debug.err file actually contained the notice that it had brought up a warning dialog about his drivers.  In email, I went back and forth with him that boiled down to this:

Me: Didn't you read the Readme where it says "We're serious, you MUST update your video drivers if they're older than a year."?

User: No. I don't have time for that.

Me: You didn't read the sticky post entitled: "Having an in-game problem? Read here for tips!" which said "You absolutely must update your video driver, if you make posts saying how the game crashes intermittently on you and it turns out you have some video driver from 2004 we'll find you! "?  And your video driver is even older than that.

User: No.

Me: You didn't heed the warning dialog that came up when you launched the game that said "Your video drivers are old.  You will experience problems if you don't update them. Your drivers are dated 9/1/2003"

User: I didn't think that applied to me. My other games don't have a problem.

This kept up over and over. And the problem is that people who don't read the Readmes or Stickies end up crowding out posts from people with legitimate issues.  Moreover, it makes it tough (at least for me) not to get jaded when someone posts yet another vague issue.  Is it yet another person who decided to rush to the forums and make another all caps post proclaiming "problems" without having even looked at the readme.   And just to be safe, we made stickies and knowledge based articles.

#3 Three people can completely change the perception of an entire forum.  I read on Penny-Arcade about companies hiring moles to go around and make their products look more well liked on forums? It would probably be more efficient to hire moles to go onto other people's forums and say negative things about someone else (That's not happening here btw, it's just a joke).  Because 3 people having problems (Real or imagined) can make enough posts and responses to make something trivial look like it's wide-spread.

Since we know the stats (how many people actually have the game) and the number of different people having a problem, we know some things I've seen are not widespread.  But you get a couple guys responding in every post or making 5 or 6 posts with the same general subject "GAME BUGS IN RELeASE!!!11" and soon you get people asking "Wow, it's buggy!"

That's not saying that the game is free of bugs. Certainly not.  But at a certain point, on a PC game, we're at the mercy of the hardware vendors and their drivers.  We can't come over to people's houses at gun point (yet) and force them to update their drivers.  We can't force handful of people running 16-bit color to change their color depth from 65,000 colors to 32bit color.  If there was an easy programmatic solution to force the color depth change (Direct3D certainly seems to have one and we're using it but on newer video drivers they seem to assume no one even runs at 16bit color anymore) we would do it.

The main thing is that I now really understand why game publishers get tempted to censor or control their forums because it's really frustrating to see a handful of people changing the whole flow. 

I also see why many publishers get accused of trying to "cover up" bugs. Because what happens is that the vocal people who don't read the readmes or the docs or even the sticky forum posts will start accusing the developer of trying to not acknowledge "a very real problem" because they don't want to admit that they just didn't bother to read the docs.

Like I said, there are definitely glitches in the game. But virtually everything I've seen (and I've been living on those forums) fall into the the trivial or the obscure.  They are things we'll fix as quickly as we can, but if you read the posts, you would think "OH man, maybe it's buggy?"

#4 Stardock Central's UI caused a lot of grief.  The biggest problem of the release in my opinion was that in Stardock Central, when you click on the game to download, it doesn't automatically expand to display the other modules. As a result, people download the game and don't download the sound, multimedia, movies, or tutorials.  I don't blame them. That's bad UI on our part and we're fixing that and putting it out as we speak.

#5 The most serious real bugs (i.e. not individual specific) I've seen are:

• If you add enough jewelry to your ship (i.e. the cosmetic addons) the game will eventually crash.  In all my game play examples and all our testing and all the testing during the beta, this never came up.  But now that real players are getting into it, some are making some super-sophisticated ships.  This is something that will be addressed in Friday's update.

• In the Metaverse, some people are reporting weird ability values.  This has been fixed (initialization issue that's only present in the release compile).

• On nVidia Geforce 6x series cards, the mouse cursor for some people disappears.  We don't have any idea on this as we can rarely get people to answer our questions on their systems (another gripe, some people seem to enjoy complaining more than solving their problem).  No idea what the cause it.  Our theory is either they're using the Omega drivers or something that doesn't support what we're doing OR they're using colored mouse cursors outside the game. 

• The GNN has a "Goto" button that can mess up the UI if the player goes deep enough in it (it was intended for people who just wanted to jump to a planet, make an adjustment and leave but people who go to the planet and then go to a bunch of other places from the planet all from within the pop up manager can overload it).  We've fixed this already.

#6 Lessons learned?

1. You can never be too prepared. Thank goodness we did the preload last week.  We never imagined the game using half a gibabit of bandwidth to bring down. Not a million years. We felt we had gone overkill on bandwidth and it was just barely enough.

2. It is impossible to find every glitch.  We had gamma testers giving us the thumbs up weeks before we sent it to manufacturing. One suggested that we were having "separation" anxiety (i.e. being afraid to let it go out there).  Sometimes bugs exist simply because someone does something you never thought of. 

3. Transparency is key.  Be open and honest with people and MOST people will respect that.  Tell people what problems you think you have and what problems you think you don't have and let people decide for themselves. Don't try to "control" the flow of information.  Stay ahead of the curve and make sure you are the one telling people what's going right and wrong.