Bugs for your buck?
I’ve just subscribed to Game Developer Magazine (www.gdmag.com) and read with interest in the September issue the article entitled Dashboard Confessional.
To sum up briefly, Midway Games outsourced their testing to a bunch of external contractors, but forgot to put in business measurements to know how well the teams were doing. The correction they took was to build a dashboard application that would analyse the numbers and tell them how each team was doing.
The thing that hit me was the numbers they were focusing on. Eseentially a team was doing well if it found a large number of bugs, there forefore had a higher “Bug for your Buck”.
I think this thinking is backwards. And it’s very common thinking, not just in the Games industry, but in the general software industry as well.
I’m interested in Agile Development, and Test driven programming, and so I believe that a test team that finds no bugs is doing just as well as a test team that finds 100 bugs, if and only if, there aren’t any bugs to find for the first team.
The games and software that we write should not have any bugs! Why should there be bugs? Bugs is an indication that the programming team have not done their job properly. Wehn you go out to dinner, and order a rare sirloin steak with potatoes and cracked pepper sauce. You expect that the kitchen will make you what you ordered. You don’t get lamb steak, with parsnips and cheese sauce because they look similar, or the cooks did their best effort.
When we write a game, we should be writing it so that there aren’t any bugs.
The best way of doing this that I know (and possibly the only way) is to write and have a comprehensive test suite. Write the tests before you program, and ensure that your tests cover every part of your code.
Testing is not the last thing you do in the devleopment lifecycle, it’s the first programming activity. Testers should be involved in design, architecture and the whole development process.
For me, the ideal Bugs for your Buck ratio would be 0. the testers worked for 4 weeks and found no bugs. Great now I can sell my software!