The first game development I did for public consumption was an add-on for Quake II. We were a team of about ten people, and I was the only software developer; everyone else was graphic artists, map designers, modelers, sound engineers, etc… all spread across the United States.
As we neared our planned launch date, I began to get moderately concerned about the fact that during the last year of development, even though every one was able to play the game on their own systems, not once did we have an actual deathmatch in which we actually played each other.
As a programmer, this was unthinkable: releasing a product specifically made for deathmatch without actually testing it for deathmatch was unimaginable, but I was outvoted by the rest of the group. They simply wanted to get the game out there so they can scream “Look! I made that!”
We released it on a Saturday morning, and less than half an hour passed before we were notified of the first “showstopper.” The bug: if two rockets collided in mid air, the game crashed instantly… and violently. It was an issue that was impossible to detect in an environment outside of deathmatch, or even in a controlled testing environment, but in the real world it happened all the time, especially in Quake II where the hit boxes of rockets were cubes two feet across.
So we pulled the release and we (or I, as the case may be) immediately went in to repair mode to release a patched version. The rest of the team finally listened to me and decided to create a test environment, but rather than do a closed alpha test they decided to invite players from the outside world to test it along with the developers.
So after I fixed all the bugs I knew of (and there were still several I didn’t know about), we hosted a deathmatch server with the “alpha” build. All eight members of the team joined in and waited until other players showed up.
Two players, anonymous denizens of the Internet that responded to our call, showed up, and through in game chat we asked them to test this by trying to take on the developers. Just for kicks, we decided to make the teams exactly as they stood: one team had all eight of us, the people that actually created the game, and the other team were these two unknowns that didn’t even know the game existed until an hour earlier.
No other “showstoppers” were found, but the two anonymous players literally decimated us. It was a rout unlike anything ever seen in the world of deathmatch. I think our death-to-kill ratio was 40:1 by the end of the match.
After a few weeks, we finally released the product… and I went my own way to work with groups that understood the concept of game testing.
On a semi-related note: one of the bugs reported was that if you uninstalled the add-on, it would also uninstall the entire Quake II directory. We had several emails from players complaining “Damn it!!! Now I have to buy Quake II and return it again!!!”