An Independent Game Developer’s Diary

“or How do I write the correct GQL to join these tables together to give me the correct output”

Yet again people are asking on the AppEngine mailing list how to write the correct sorts of joins to join some tables together.

Looking at the design, it was clearly a great database design, a product table, a customer table, and a many to many join table with an attribute to qualify the join. Exactly the sort of tables that you were taught to write in your CompSci degree or courses, or learnt from experience over years of working with databases.

The problem is that BigTable is not a relational database. The Google team has tried as hard as possible to emphasise this, but they didn’t really provide enough examples that people really got it clear.

BigTable is not a relational database, now go write that out a hundred times, and when your done come back and finish reading.

When it comes to BigTable you want to design your database not around “What bits of data should be grouped together”, but around the principle, “What bits of data will I need to read concurrently”.
In this case, what you want to get out is
Purchase information (Order, Date), Customer information (Name, Country, …), Product information (Name, Code, …).

Therefore your table should be.
Purchase:
Customer Name
Customer Country
Product Code
Product Name
Purchase Order Number
Date Of Order

Now your query becomes

Purchase.all().filter("customer_name =",customer).filter("product_name =", product).fetch(100)

Easy huh? It’s all one big table.
Say you want to show a customer what items they ordered for a given purchase order…

Purchase.all().filter("purchase_order =",order_num).fetch(100)

Updating becomes a little harder, but is still pretty easy.
The question then comes, how do I update a product name, if I change my widgets from Acme Widget to “Acme Superwidget”?
Well firstly, really, how often do you do that? I bet it’s a number of orders of magnitude less often than you execute the queries above.

If you must, it is something like:

items = Purchase.all().filter("product_name =",product).fetch(1000)
for item in items:
item.product_name = new_name
item.put()

Yes that last part is a bit nasty, if GQL supported updates it would be nice, but writing the updates in python is hardly taxing.

I hope that helps someone

So I’ve been doing this for almost a week, and I’ve got another version up at shipwreck.mibgames.co.uk.  It’s still not really playable, there’s certainly not a game there yet, but it’s a lot better than it was.

• Now when you create a character, you get the generic stats, and can’t edit them
• Your characters location is randomly set to be a beach location along the north coast of the island at creation
• You can now see a map of the areas around your character (using some free graphics I found online and threw together in 5 minutes, hence the mismatched sizes)
• You can click on a map location to travel there, assuming that it’s possible (you can’t walk on the sea)
• All new code uploaded to the google code project

So, I’m feeling slightly proud, or was until I started writing my todo list, here’s the main features waiting to be done

• Character creation should be fleshed out, choose a profession
• Action points based on time, and moving around the map should cost action points
• See other characters on the map, and send them messages
• Character advancement commands - (learn skills, practice skills)
• Items - Allow items to be found, and picked up
• Monsters - Allow monsters to be met and fought.
• Interzone travel - Currently you can’t travel from zone 1 to zone 2, this should be fixed.
• Finish drawing the map, my laptop power ran out last night, and the mpa was only half started.
• Better / More graphics, including Terrain tiles, characters, monsters

Phew!

You’ll also notice if you look at the code that there are no tests anywhere.  This is against my normal coding practice, but the django test suite refuses to work under AppEngine (the database layer is required to run the django tests) and I haven’t got around to writing proper tests for anything yet.  The key classes to write tests for are the helpers.  That would probably also help me split a lot of code out from the views and into more helper style methods.

I got a response from google about my weird unexpected behaviour when calling filter in my application.  It turns out to be expected behaviour, but is almost guaranteed to trip up a new developer.

The behaviour is that if you use a filter string like “property=” you get a None back rather than the data that you expected from the BigTable datastore.

The reason is that ‘=’ is a valid character for a property name, so the parser is unable to tell if your filter means the property called “property” = or the property called “property=”.

The end result is that you must put a space between the property name and the equals sign in order to  get valid results

Example behavour:

from google.appengine.ext import db
class A(db.Model):
b = db.IntegerProperty()
A(b=1).put()

#Works
print A.all().filter("b =").get()

# Fails, returns None
print A.all().filter("b=").get()

# Throws exception
print A.all().filter("b= ").get()

So my host just started working this morning, so you can see the first stab at the app at http://shipwreck.mibgames.co.uk

I still haven’t quite got the hang of the developer accounts, but I’m persevering with my gmail account, although I would have liked a way to subscribe to the google group with my mibgames account, as I actually read that one daily, whereas gmail I don’t normally make a habit of reading. Although I’ve also apparently managed to get duplicate emails being sent to my personal email address, but haven’t worked out why yet. I’m sure I’ll work it out.

[Edit: Found out why, I had setup my personal account to download email from my gmail account via POP3 ages ago and hadn't turned it off.]

I also had my first couple of weird issues. I had created a django template, called base.html, which defined the base layout (or will do eventually), and was creating a template called create_character.html which inherited from the base template. I defined my title and main blocks in my template, and filled them in, and when I hit the template I got the weird error, TemplateSyntaxError that appeared to say that {% block main %} had a syntax error, and that it couldn’t read the expected next character.

I scratched my head a bit, tried renaming the blocks, tried rewriting it, to make sure I hadn’t done something really stupid. Eventually, I moved the blocks around, moving the title block to the end, and the main block to before the title, and it suddenly started working. Changing it back now doesn’t break it anymore, so I’ve no idea what caused it. Very weird.

My second issue is to do with getting stuff out of the BigTable, but first I had to get something in, and this is where I met my second favourite feature so far. I created a model for the player’s character, containing some basic stats, skills, and admin values. Having created the model, I want to try to be able to create one, save it to the BigTable, and then get it out again.

But the easiest way of creating a character without needing to generate random numbers, and do weird things, is to create a form with all of the relevant fields, and I can manually enter the values, press save and voila, it would save. So I set about creating a Django form for my model object. Having entered the 17 lines of attributes, run a few vim commands to replace db.StringProperty with forms.CharField and stuff, I thought, there must be an easier way than this.

And there is, in google.appengine.ext.db there is a module called djangoforms. Instead of my 17 lines of code, and the hiddeous thought of writing some manual code to hookup the form.clean_data['strength'] to character.strength, you can instead use the following lines of code

from google.appengine.ext.db import djangoforms

class CreateCharacterForm(djangoforms.ModelForm):
class Meta:
model = Character
exclude = ['location', 'health', 'armour', 'owner']

Yup, thats it, create a djangoforms.ModelForm, and give it a Meta class that specifies which model object you want it to associate with, and it does. In your view you just do

form = CreateCharacterForm(request.POST)
form.save()

and it does… unless you’ve declared anything as required, and excluded it from the form at the same time. Trying to wrok out how to get access to the model object, set it’s owner value before saving it was quite hard, until I noticed a parameter on the save method, commit=True. Checking out the source, it seemed true, if you give the keyword parameter commit and set it to False, the form.save method simply sets up your model object, fills it from the form and returns it to you, unsaved. I can then do

character = form.save(commit=False)
character.owner = users.get_current_user()
character.put()

And bobs your transvestite uncle who keeps turning up to family gatherings even though nobody has invited him.

My only problem now is that having executed this once, the object should exist and be in the datastore. But my code to get it back out, doesn’t seem to work.

I’m probably doing something really stupid, but from what I can read from the docs, this code should work.

character = Character.all().filter('owner=', users.get_current_user()).fetch(1)

I’ll work on it tomorrow and see what I can get working then.

Oh last feature, and this is my favourite feature so far. While looking to try to work out if my data was in the datastore, I found the dev_appserver admin interface. It doesn’t provide you with the logs, CPU cycles used or bandwidth obviously, but it does provide the facility to dynamically examine the datastore, and an interactive python interpreter, so you can just import your model and execute query methods to your hearts content.

Yes! My first django app, the framework for the MMO game has been created and uploaded.

Props goes to the person responsible for me getting an account, now all I have to do is blog about it as I go along. Given my level of excitement that shouldn’t be a problem for a few months at least.

Anyway, slightly pre-empting my blogpost tomorrow about settings and styles of MMO, I’ve created the app with the name shipwreck, and you can see the working code at here

My first thoughts, this is cool! I like it.

It’s a bit of a pain to create a django app, you end up creating a wrapping app that encases the django app, so my directory structure has shipwreck/shipwreck/main to get to the django app, inside the django project, inside the wrapper.

Secondly, I’ve had a bit of difficulty with my google apps for domains account. My gmail account is the one I got an AppEngine account for, but I’d like to be able to host the project at shipwreck.mibgames.co.uk, and develop and upload as my mibgames.co.uk account
Rather suprisingly to me, the admin interface actually supports this, I was able to add my mibgames.co.uk email address as a developer, and the email had a special link to confirm if mibgames.co.uk was an apps for domains domain name.
Secondly, you can request an external domain name for your app, requiring a simple CNAME change in DNS, which I did, but for some reason it wont activate yet.

Part of the problem is probably related to the same issues I’ve had before, which is logging into Google services, often the account I get logged in with can appear to be random as to whether it is my apps for domains account, or my gmail account as I have authentication saved on my development machines for both of them.

I’m sure those niggles will iron themselves out as I start to play.

Overview, so far, putting django, with templates, into an app and uploading it, very easy. Next, creating a database model and building the first couple of screens!