An Independent Game Developer’s Diary

Simon Stewart and Julian Simpson had a public discussion about the benefits or not of developing a build pipeline.

As you may be able to read from my headline, I’m a huge fan of CI and build pipelines!
Actually, I am a big fan of continuous integration and cruise control where it helps the project. However we can sometimes get over enamoured of continuous integration builds.
Many of the standard XP texts explain that you need CI, and in a way they are right. The benefits that automated tests give are enormous. You have the safety fallback that if you manage to accidentally affect the behaviour of the current system, and you can actively prevent against reintroducing bugs.

But the cost of doing so is that as the project gets bigger, your tests may start to take longer to run. At this point there are many technical solutions to the problem, of which the build pipeline is one of many. However the problem is not a technical problem.

Lets break it down shall we?

The problem is that running the integration tests take too long.
As with any agile system, we can decide to either
a) throw more money at it (read developer time)
b) throw more time at it (read, leave it alone)
c) reduce the scope (read, remove tests)

Yes you read that last one right, you can remove tests. It’s possible, and although you may feel dirty for suggesting it, you have to look at your tests and decide if they are giving you value.

So how does removing tests help us? well in the case of a continuous integration system, the tests are required tests that show that the system is in a workable state. Aren’t they? or are they?
Certainly at our company we use test driven development, so for every story we will have a number of “fast” tests, these are unit tests, run through jUnit, and some database tests which are also run through jUnit, but actually access a live database. Secondly we often write a couple of selenium tests, these script a web-browser session, and test that the features asked for in the user story.
However after the project has been live for nearly a year, and we have delivered hundreds of stories, we now have a lot of tests. Many of the selenium tests test features that we no longer touch, and don’t provide useful feedback, all they do is absorb development time.
So maybe our next step should be to remove those tests that don’t provide any value.

Of course if we remove the tests, what happens if we want to work in those areas?
Well, I don’t recommend actually removing the tests. Currently our tests take at least 30 minutes to run, when run in a CI environment, on every checkin, that is an inexcusably long time, however, run by the QA department, by hand, when they are ready to accept a build into the full QA testing process (which may take a few days), they are worth running. By moving the tests, we speed up the pipeline of doom, even down to the point it may be possible to remove the pipeline and return to a single CI build.

Yes, I am still alive!

The last few months have been difficult for me, and there have been a bunch of changes, so I haven’t made time to blog. (Note, you always have time, you just don’t make time!).

I am going to try to correct that and start blogging at least once a week again, although due to my change in circumstances the content of the blog posts may drift from games occasionally, although the direction they will drift will probably cover agile management, project management and coding, so  they should still stay interesting.

So what is the big change then?  Well, due to a severe lack of funds, and the increase in living costs incurred from moving into a nice house instead of the bedroom that we were living in, I had to start searching for a job.  And I found one, a good one, working for this little provincial newspaper called The Guardian, you may have heard of it.  I am working with the web development team building the software and website for the new look guardian website.

It’s an exciting opportunity, because although it’s totally unrelated to the games industry, it is related to my second love, which is agile software development.  I’m getting a chance to work in a fully XP environment, pair programming, unit testing and so forth.

In the meantime, my business carries on, the web hosting business is making enough money to tide itself over, and although because of the stress of starting a new job has given me a pause in working on the game I am continuing to work on the game in the background.
Anyway, a blog post on accepting and learning from failure will be coming next…

I’ve just subscribed to Game Developer Magazine (www.gdmag.com) and read with interest in the September issue the article entitled Dashboard Confessional.

To sum up briefly, Midway Games outsourced their testing to a bunch of external contractors, but forgot to put in business measurements to know how well the teams were doing. The correction they took was to build a dashboard application that would analyse the numbers and tell them how each team was doing.

The thing that hit me was the numbers they were focusing on. Eseentially a team was doing well if it found a large number of bugs, there forefore had a higher “Bug for your Buck”.

I think this thinking is backwards. And it’s very common thinking, not just in the Games industry, but in the general software industry as well.

I’m interested in Agile Development, and Test driven programming, and so I believe that a test team that finds no bugs is doing just as well as a test team that finds 100 bugs, if and only if, there aren’t any bugs to find for the first team.

The games and software that we write should not have any bugs! Why should there be bugs? Bugs is an indication that the programming team have not done their job properly. Wehn you go out to dinner, and order a rare sirloin steak with potatoes and cracked pepper sauce. You expect that the kitchen will make you what you ordered. You don’t get lamb steak, with parsnips and cheese sauce because they look similar, or the cooks did their best effort.

When we write a game, we should be writing it so that there aren’t any bugs.

The best way of doing this that I know (and possibly the only way) is to write and have a comprehensive test suite. Write the tests before you program, and ensure that your tests cover every part of your code.

Testing is not the last thing you do in the devleopment lifecycle, it’s the first programming activity. Testers should be involved in design, architecture and the whole development process.

For me, the ideal Bugs for your Buck ratio would be 0. the testers worked for 4 weeks and found no bugs. Great now I can sell my software!

I wonder how other indies manage their development time?

What I mean is how do they decide what needs to be done, whether they are on track for doing the correct tasks, and how do they choose what to do from day to day.

I know there have been discussions on IndieGamer Development forums about staying motivated. Now only having only started two weeks ago, I’m not having that problem yet, but I wonder if good planning might help in the future.

The method I’ve been using is a combination of eXtreme Programming and Scrum. Essentially I am working to week long iterations. At the end of each iteration, I aim to have a working game (or game like system), and I’m picking tasks to do within each iteration. I have a white board (actually a very cool glass whiteboard, only £8 from Ikea!) upon which I’ve written up the list of tasks.

(click to enlarge)

I have planned out some large scale tasks, aiming for about two a week for the next 12 weeks (thats the stuff to the left), and is very rough. However each week, I breakdown the tasks into much smaller tasks. As can be seen (or not) in this image, I had two large tasks this week, Scrolling and Turns.

I broke scrolling into the following mini-tasks

• Create Viewport (4 hours)
• Efficiency Algorithm (4 hours)
• Edge Detection (2 hours)
• Refactor Locations (8 hours)

On Monday mornings I take half an hour to an hour to break this weeks tasks down like this, and I write them both on the board, and in a spreadsheet.

E very morning I update the board, and for each task I do one of three things

1. Task hasn’t changed, leave it as is
2. Task has been completed, put a line through it
3. Task is partially done, I change the hours estimate to record how long there is left on this task

I then add a new column to the spreadsheet after that, with the new column of numbers. This means I can graph the total hours estimated going down and see if I’ve had a bad day.

I like this system, it’s very similar to how I worked at my previous workplace, with the whole team, and I’ve found it takes me about 10 minutes a morning to update, but gives me a good feeling come about wednesday when I cna look at the board and see half of the tasks with lines through them.

So how do you do it?