Homebrew and MySQL server

So, I got a fresh new macbook last week and decided to use homebrew for my MySQL server installation. I’ve used homebrew for a while and love it, but I had installed MySQL from a package before I knew about homebrew (and maybe before it existed). Anyway, here’s the problem I ran into:

Everything installed just splendidly using brew install mysql, but I have a particularly heavy query on a dev site and the server was responding with “MySQL server has gone away.” No errors were showing up in the .err file (/usr/local/var/mysql) until I used mysql.server start --log-warnings=2. It turned out this was because the max_allowed_packet was too small, so I started the server with mysql.server start --max-allowed-packet=16M and I was golden.

So, I went to put this in my my.cnf file… but there wasn’t one!! I couldn’t find any configurations anywhere! Seemed really weird until I realized that mysql.server was actually running msqld with all the options passed in. So, I tried creating an /etc/my.cnf file and then ~/.my.cnf, but they didn’t work. I kept getting errors like:

ERROR! MySQL server PID file could not be found!
or
ERROR! The server quit without updating PID file
or
ERROR! The server quit without updating PID file (/usr/local/var/mysql/cambiodos.local.pid)

I think this may have had something to do with my setting the mysql user to myself. Shit, it seemed to make sense at the time. So, with “ERROR” ringing in my head and my blood pressure rising, I read on the MySQL site somewhere that ~/.my.ini was an alternative to ~/.my.cnf. For some reason it made sense to me to try it and it worked.

I’m guessing this has something to do with the fact that my.cnf is a configuration file and my.ini is just settings? So by using the .ini file I was able to simply override settings, not the absolute configuration, but that’s just a guess. Anyway, I’m happily running mysql with the following ~/.my.ini file:

[mysqld]
max_allowed_packet = 16M
log_warnings = 2
default-character-set = utf8

Better way to use Django’s SelectMultiple in Google App Engine for a ListProperty

So my last post was a quick hack attempt at this, but I have come up with a much cleaner method of getting a GAE ListProperty field to work with Django Forms (newforms). You simply have to create a new form field called ListPropertyChoice:

from django.newforms.fields import MultipleChoiceField
from appengine_django.models import BaseModel

class ListPropertyChoice(MultipleChoiceField):

    def clean(self, value):
        """ extending the clean method to work with GAE keys """
        new_value = super(ListPropertyChoice, self).clean(value)
        key_list = []
        for k in new_value:
            key_list.append(BaseModel.get(k).key())
        return key_list

You can then use this in your forms:

...
from fields import ListPropertyChoice

class Form(djangoforms.ModelForm):
    my_model = ListPropertyChoice(
                        widget=forms.CheckboxSelectMultiple(),
                        choices=[(m.key(), m.name) for m in db.Query(MyModel)]
                    )
    class Meta:
        model = ParentModel

Obviously you can use both the SelectMultiple widget and the CheckboxSelectMultiple widget and this same process could be easily duplicated for ChoiceField:

class GAEChoiceField(ChoiceField):
    def clean(self, value):
        """ extending the clean method to work with GAE keys """
        value = super(GAEChoiceField, self).clean(value)
        return BaseModel.get(value).key()

Django ImageField and FileField dynamic upload path in Newforms Admin

I’ve been using Scott’s CustomImageField for quite some time, but recently it stopped working when I moved to Newforms Admin. I wasn’t really sure why this was, but decided to build a quick workaround for now so I can continue to use the newforms-admin.

Scott’s Field used the post_init and pre_save signals, which is probably the right way… Since this wasn’t working anymore, I decided to use the post_save signal. This way the image or file would be saved to a temporary directory initially and then moved to the appropriate directory afterwards.

Here is how the field would be defined in the model:

    class Image(models.Model):
	file = CustomImageField(use_key=True, upload_to='tmp')

use_key and upload_to are optional. use_key defaults to False. If it is True then the id of the instance will be used as a prefix for the new file as there is the potential for overwriting now that we are moving the file. upload_to will simply define the temporary directory to upload the files to initially.

Below is the code that defines the CustomImageField. I have this in a file called “custom_fields.py.”

from django.db.models import ImageField, FileField, signals
from django.dispatch import dispatcher
from django.conf import settings
import shutil, os, glob, re
from distutils.dir_util import mkpath

class CustomImageField(ImageField):
    """Allows model instance to specify upload_to dynamically.

    Model class should have a method like:

        def get_upload_to(self, attname):
            return 'path/to/%d' % self.id

    Based closely on: http://scottbarnham.com/blog/2007/07/31/uploading-images-to-a-dynamic-path-with-django/
    """
    def __init__(self, *args, **kwargs):
        if not 'upload_to' in kwargs:
            kwargs['upload_to'] = 'tmp'
        self.use_key = kwargs.get('use_key', False)
        if 'use_key' in kwargs:
            del(kwargs['use_key'])
        super(CustomImageField, self).__init__(*args, **kwargs)

    def contribute_to_class(self, cls, name):
        """Hook up events so we can access the instance."""
        super(CustomImageField, self).contribute_to_class(cls, name)
        dispatcher.connect(self._move_image, signal=signals.post_save, sender=cls)

    def _move_image(self, instance=None):
        """
            Function to move the temporarily uploaded image to a more suitable directory 
            using the model's get_upload_to() method.
        """
        if hasattr(instance, 'get_upload_to'):
            src = getattr(instance, self.attname)
            if src:
                m = re.match(r"%s/(.*)" % self.upload_to, src)
                if m:
                    if self.use_key:
                        dst = "%s/%d_%s" % (instance.get_upload_to(self.attname), instance.id, m.groups()[0])
                    else:
                        dst = "%s/%s" % (instance.get_upload_to(self.attname), m.groups()[0])
                    basedir = "%s%s/" % (settings.MEDIA_ROOT, os.path.dirname(dst))
                    mkpath(basedir)
                    shutil.move("%s%s" % (settings.MEDIA_ROOT, src),"%s%s" % (settings.MEDIA_ROOT, dst))
                    setattr(instance, self.attname, dst)
                    instance.save()

    def db_type(self):
        """Required by Django for ORM."""
        return 'varchar(100)'

SelectMultiple for a Google App Engine ListProperty using Django Forms

***EDIT: I recently posted a better way to do this.

I just started with Google App Engine the other day. Man, who could resist a free server environment?

Being a long time Django developer (one year), I was really excited about it because it “runs django”. Now this isn’t really the case because of BigTable, which isn’t a real relational database. However, if you forgo the Django ORM and use GAE models, you can still work with the other things that make Django so awesome: templates and forms. AND, the google models are so similar to Django that you’ll barely even notice the difference.

Ok, this isn’t a full on review GAE with Django, so I’ll cut right to the chase. I’m sure you found this page when you were trying to use django forms to handle the ListProperty model field in your HTML forms. The problem is that things like ModelChoiceField and ModelMultipleChoiceField won’t work because they rely on the Django ORM. For that matter, ChoiceField and MultipleChoiceField won’t work because they return a list of strings, that although they can be the string for a db.key(), they won’t be the Key objects that your new models expect for ListProperty.

So, to get around this I just adjusted the MultipleChoiceField a bit so that it returns the list chosen fields as Key objects:

#MODELS
class Toping(BaseModel):
    name = db.StringProperty(required=True)

class Pizza(BaseModel):
    name = db.StringProperty(required=True)
    toppings = db.ListProperty(db.Key)

    def get_topings(self):
        # This returns the models themselves, not just the keys that are stored in toppings
        return Topping.get(self.toppings)

#FORMS
class PizzaForm(djangoforms.ModelForm):
    toppings = ListPropertyChoice(
                        widget=forms.CheckboxSelectMultiple(),
                        choices=[(rt.key(), rt.name) for rt in db.Query(Topping)]
                    )
    class Meta:
        model = Pizza

#CUSTOM FIELD
from django.newforms.util import ErrorList, ValidationError
from django.newforms.fields import ChoiceField
from django.newforms.widgets import MultipleHiddenInput, SelectMultiple
from appengine_django.models import BaseModel

from appengine_django.serializer.python import smart_unicode
from django.utils.translation import gettext

class ListPropertyChoice(ChoiceField):
    hidden_widget = MultipleHiddenInput

    def __init__(self, choices=(), required=True, widget=SelectMultiple, label=None, initial=None, help_text=None):
        super(ListPropertyChoice, self).__init__(choices, required, widget, label, initial, help_text)

    def clean(self, value):
        """
        Validates that the input is a list or tuple.
        """
        if self.required and not value:
            raise ValidationError(gettext(u'This field is required.'))
        elif not self.required and not value:
            return []
        if not isinstance(value, (list, tuple)):
            raise ValidationError(gettext(u'Enter a list of values.'))
        new_value = []
        for val in value:
            val = smart_unicode(val)
            new_value.append(val)
        # Validate that each value in the value list is in self.choices.
        valid_values = set([smart_unicode(k) for k, v in self.choices])
        for val in new_value:
            if val not in valid_values:
                raise ValidationError(gettext(u'Select a valid choice. %s is not one of the available choices.') % val)
        # These are the only changes to the django MultipleChoice Field
        # we just convert the list of strings to a list of keys
        key_list = []
        for k in new_value:
            key_list.append(BaseModel.get(k).key())
        return key_list

Now for a quick expaination… The ListPropertyChoice class itself is really just the MultipleChoiceField from Django V0.96. I had to use this because I couldn’t get the latest svn version to work w/ App Engine due to some SafeUnicode issues I haven’t looked into yet. The only real changes to that class are the last four lines which return the list of keys instead of key-strings from the form.

The only other thing that might need explanation is the get_toppings() method that I added to the Pizza model. This is just a shortcut to access the models instead of the keys, as they’re stored in the ListProperty.

Bulk Add/Replace from Command Line

I had to do this like 5 times today, so I thought I’d put it out there (so I can reference it).

Note: you might want to backup your stuff, just in case you screw up your regular expression.

Basically, I have a bunch of files that need to have changes, such as email address, or include paths.

So, for example, in one I had to change a bunch of email addresses so I used `sed`. To do this on one file you would simply run the sed command directly:
sed -e s/info@olddomain.com/info@newdomain.com/g -i file.html
-e tags the regular expression you’re using
-i means we’re editing the files inline (not outputting the results to standard out)
Don’t forget g, because this will ensure that the expression is greedy and will find all the instances of this expression.

Of course, you probably want to run this on a bunch of files, so you would use `find`, because it’s awesome:
find -name "*.html" -exec sed -e s/info@olddomain.com/info@newdomain.com/g -i {} \;

EDIT: I have found that Macs have an issue with the inline command (-i). You need to use `-i ”` to give the inline an empty extension. So, the command above would be:

find -name “*.html” -exec sed -e s/info@olddomain.com/info@newdomain.com/g -i ” {} \;

Starting a new Django Project with Versioning

Having an efficient working environment can be essential to the success of any project and can save hours, if not days of extra work in the long run. So, I’ve put together a little “getting started guide” to putting the full package in place with Django, SVN, and Trac.

Create your django project anywhere

~/django/projects$ ../src/trunk/django/bin/django-admin.py startproject myProject

Set up your SVN
This can be done either on your local machine or on another server. I like /var/svn/repos/myProject/.

$ svnadmin create /var/svn/repos/myProject

I use svnserve to serve my svn so I update the permissions in /var/svn/repos/myProject/conf/svnserve.conf:

[general]
anon-access = none
auth-access = write
password-db = passwd

/var/svn/repos/myProject/conf/passwd

[users]
user = passwd

(re)Start the svnserve daemon

svnserve -d -r /var/svn/repos

Finally, we’re ready to check in the project tree

$ svn import myProject file:///var/svn/repos/myProject/trunk/myProject -m "Initial import"

Now just import it into Eclipse and your good to go! ;)

Oh, and you might want to read about excluding .pyc files in svn.

Compress a PDF with pdftk

UPDATED (9/7/11): The package is not pdftk, but Ghostscript, and it’s easily installable through homebrew on a mac.

Ok, so I don’t know why this works, but it does and I haven’t seen any decrease in quality. I suspect converting from PDF to PS and back just optimizes the PDF.
Requirements

The Process

  1. Convert your PDF to PS (this creates a large file
  2. Convert the new PS back to a PDF


pdf2ps large.pdf very_large.ps
ps2pdf very_large.ps small.pdf

Results
large.pdf : 6.3MB
very_large.ps : 53.4MB
small.pdf : 2.4MB
Looks like pretty good compression to me.

If anyone knows why this works so well, please let me know.