Category Archives: webdev

Better way to use Django’s SelectMultiple in Google App Engine for a ListProperty

So my last post was a quick hack attempt at this, but I have come up with a much cleaner method of getting a GAE ListProperty field to work with Django Forms (newforms). You simply have to create a new form field called ListPropertyChoice:

from django.newforms.fields import MultipleChoiceField
from appengine_django.models import BaseModel

class ListPropertyChoice(MultipleChoiceField):

    def clean(self, value):
        """ extending the clean method to work with GAE keys """
        new_value = super(ListPropertyChoice, self).clean(value)
        key_list = []
        for k in new_value:
        return key_list

You can then use this in your forms:

from fields import ListPropertyChoice

class Form(djangoforms.ModelForm):
    my_model = ListPropertyChoice(
                        choices=[(m.key(), for m in db.Query(MyModel)]
    class Meta:
        model = ParentModel

Obviously you can use both the SelectMultiple widget and the CheckboxSelectMultiple widget and this same process could be easily duplicated for ChoiceField:

class GAEChoiceField(ChoiceField):
    def clean(self, value):
        """ extending the clean method to work with GAE keys """
        value = super(GAEChoiceField, self).clean(value)
        return BaseModel.get(value).key()

SelectMultiple for a Google App Engine ListProperty using Django Forms

***EDIT: I recently posted a better way to do this.

I just started with Google App Engine the other day. Man, who could resist a free server environment?

Being a long time Django developer (one year), I was really excited about it because it “runs django”. Now this isn’t really the case because of BigTable, which isn’t a real relational database. However, if you forgo the Django ORM and use GAE models, you can still work with the other things that make Django so awesome: templates and forms. AND, the google models are so similar to Django that you’ll barely even notice the difference.

Ok, this isn’t a full on review GAE with Django, so I’ll cut right to the chase. I’m sure you found this page when you were trying to use django forms to handle the ListProperty model field in your HTML forms. The problem is that things like ModelChoiceField and ModelMultipleChoiceField won’t work because they rely on the Django ORM. For that matter, ChoiceField and MultipleChoiceField won’t work because they return a list of strings, that although they can be the string for a db.key(), they won’t be the Key objects that your new models expect for ListProperty.

So, to get around this I just adjusted the MultipleChoiceField a bit so that it returns the list chosen fields as Key objects:

class Toping(BaseModel):
    name = db.StringProperty(required=True)

class Pizza(BaseModel):
    name = db.StringProperty(required=True)
    toppings = db.ListProperty(db.Key)

    def get_topings(self):
        # This returns the models themselves, not just the keys that are stored in toppings
        return Topping.get(self.toppings)

class PizzaForm(djangoforms.ModelForm):
    toppings = ListPropertyChoice(
                        choices=[(rt.key(), for rt in db.Query(Topping)]
    class Meta:
        model = Pizza

from django.newforms.util import ErrorList, ValidationError
from django.newforms.fields import ChoiceField
from django.newforms.widgets import MultipleHiddenInput, SelectMultiple
from appengine_django.models import BaseModel

from appengine_django.serializer.python import smart_unicode
from django.utils.translation import gettext

class ListPropertyChoice(ChoiceField):
    hidden_widget = MultipleHiddenInput

    def __init__(self, choices=(), required=True, widget=SelectMultiple, label=None, initial=None, help_text=None):
        super(ListPropertyChoice, self).__init__(choices, required, widget, label, initial, help_text)

    def clean(self, value):
        Validates that the input is a list or tuple.
        if self.required and not value:
            raise ValidationError(gettext(u'This field is required.'))
        elif not self.required and not value:
            return []
        if not isinstance(value, (list, tuple)):
            raise ValidationError(gettext(u'Enter a list of values.'))
        new_value = []
        for val in value:
            val = smart_unicode(val)
        # Validate that each value in the value list is in self.choices.
        valid_values = set([smart_unicode(k) for k, v in self.choices])
        for val in new_value:
            if val not in valid_values:
                raise ValidationError(gettext(u'Select a valid choice. %s is not one of the available choices.') % val)
        # These are the only changes to the django MultipleChoice Field
        # we just convert the list of strings to a list of keys
        key_list = []
        for k in new_value:
        return key_list

Now for a quick expaination… The ListPropertyChoice class itself is really just the MultipleChoiceField from Django V0.96. I had to use this because I couldn’t get the latest svn version to work w/ App Engine due to some SafeUnicode issues I haven’t looked into yet. The only real changes to that class are the last four lines which return the list of keys instead of key-strings from the form.

The only other thing that might need explanation is the get_toppings() method that I added to the Pizza model. This is just a shortcut to access the models instead of the keys, as they’re stored in the ListProperty.

Starting a new Django Project with Versioning

Having an efficient working environment can be essential to the success of any project and can save hours, if not days of extra work in the long run. So, I’ve put together a little “getting started guide” to putting the full package in place with Django, SVN, and Trac.

Create your django project anywhere

~/django/projects$ ../src/trunk/django/bin/ startproject myProject

Set up your SVN
This can be done either on your local machine or on another server. I like /var/svn/repos/myProject/.

$ svnadmin create /var/svn/repos/myProject

I use svnserve to serve my svn so I update the permissions in /var/svn/repos/myProject/conf/svnserve.conf:

anon-access = none
auth-access = write
password-db = passwd


user = passwd

(re)Start the svnserve daemon

svnserve -d -r /var/svn/repos

Finally, we’re ready to check in the project tree

$ svn import myProject file:///var/svn/repos/myProject/trunk/myProject -m "Initial import"

Now just import it into Eclipse and your good to go! 😉

Oh, and you might want to read about excluding .pyc files in svn.

Ignore .pyc files in Subversion

Subversion allows you to exclude certain files using regular expressions. This needs to be applied to each directory individually. So, if you add a new directory I think you will need to run the command again. To apply the svn:ignore property to your project you just need to do two things:

Create a file (.svnignore) with the regular expressions


Run svn propset on the project

svn -R propset svn:ignore -F .svnignore .

-R indicates that you are doing this recursively so all the directories ignore these.
-F indicates the file we just created with the regular expressions.

EDIT 5/27/08:
If you’ve already got yourself into the mess of versioning your compiled files, then you might want to do a few things.
1) Revert all the .pyc files (you can always recompile them). You don’t want there to be any local changes when you try to delete them:
find -name "*.pyc" -exec svn revert {} \;
2) Delete all the .pyc files using `svn delete`:
find -name "*.pyc" -exec svn delete {} \;
3) Finally recompile and re-run the svn ignore code above

Quick image resizing with python

I typically build automatic resizing into any django models that use ImageField by overriding the save function like so:

def save(self):
super(GalleryImage, self).save() # Call the "real" save() method
if self.image:
# now change the size of the image
path = settings.MEDIA_ROOT + self.image
img1 =
img2 =

size = 600,600
img1.thumbnail(size, PIL.ANTIALIAS)

# create the thumbnail
size = 150, 150
img2.thumbnail(size, PIL.ANTIALIAS)

def get_thumbnail_path(self):
path = settings.MEDIA_ROOT + self.image
return self.convert_path_to_thumbnail(path)

def get_thumbnail_url(self):
path = self.get_image_url()
return self.convert_path_to_thumbnail(path)

def convert_path_to_thumbnail(self, path):

basedir = os.path.dirname(path) + '/'
base, ext = os.path.splitext(os.path.basename(path))

# make thumbnail filename
th_name = base + "_tn"
th_name += ext
return urlparse.urljoin(basedir, th_name)

This ensures that all my images conform to the sizes I need. However, even if the server is automatically resizing the images for me, I don’t want to spend hours uploading huge images only to have them resized. It’s just a waste of bandwidth and time. So, I’ve written a quick python script that resizes all the images in a given directory and places the copies in a new directory called resized:

#! /usr/bin/env python

Resize all the jpg images in a directory
All images will be placed inside a directory called "resized"
This directory must exist
Usage: ./

import PIL.Image as PIL
import re, os, sys, urlparse

SIZE = 600,600
JPG = re.compile(".*\.(jpg|jpeg)", re.IGNORECASE)

def get_new_path(path):
basedir = os.path.dirname(path) + '/resized/'
base, ext = os.path.splitext(os.path.basename(path))
file = base + ext

return urlparse.urljoin(basedir, file)

image_dir = sys.argv[1]
print "You must specify a directory"

files = os.listdir(image_dir)
for file in files:
if JPG.match(file):
f = image_dir.rstrip("/") + "/" + file
print f
img =
img.thumbnail(SIZE, PIL.ANTIALIAS)

Nginx SSL passphrase at startup

I’m using nginx to serve all my media files and it is also a proxy to my apache server. I’ve been using SSL encryption for specific sections of the site, using nginx’s built in SSL support:

server {

listen 443;

access_log /var/log/nginx/register/host.access.log;

ssl on;
ssl_certificate /home/jamstooks/ssl_certs/2008_cert/;
ssl_certificate_key /home/jamstooks/ssl_certs/2008_cert/;

# Register
location ^~ /register/ {
proxy_redirect off;

proxy_set_header Host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

client_max_body_size 10m;
client_body_buffer_size 128k;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

# Redirect everything else
location / {
rewrite ^(.*)$1 permanent;


When I boot up Nginx it requests the passphrase for the encrypted certificate key. This is a huge problem though when there are unexpected shutdowns because the Nginx process won’t restart.

However, the problem is not with Nginx, but with the certificate itself. Because it is encrypted, Nginx can’t use it unless it until it has the pass-phrase. So, the easiest way to solve this is to provide Nginx with a decrypted version of the certificate key. The only issue is that you need to tie down the permissions on the file so that no one can access it at use it to impersonate you.

Apache details the process here:

# Remove the encryption from the RSA private key (while keeping a backup copy of the original file):

$ cp server.key
$ openssl rsa -in -out server.key

# Make sure the server.key file is only readable by root:

$ chmod 400 server.key

ModelMultipleChoiceField save fails

I’ve been banging my head today trying to get a MultipleChoiceField or a ModelMultipleChoiceField to save correctly. I was finding that no matter which widgets I used, or which field types I was using, it was failing. I wouldn’t get any errors, and all the data would be saved except for my ManyToMany relationship.

So, after much ado creating custom widgets and scouring the web for advice, I found that I should have just read the ModelForm Documentation. There it was, under ‘The save() method”:

Another side effect of using commit=False is seen when your model has a many-to-many relation with another model. If your model has a many-to-many relation and you specify commit=False when you save a form, Django cannot immediately save the form data for the many-to-many relation. This is because it isn’t possible to save many-to-many data for an instance until the instance exists in the database.

I had been using save(commit=false) and hadn’t been using save_m2m(). That’ll teach me to read the documentation!