Ignore .pyc files in Subversion

Subversion allows you to exclude certain files using regular expressions. This needs to be applied to each directory individually. So, if you add a new directory I think you will need to run the command again. To apply the svn:ignore property to your project you just need to do two things:

Create a file (.svnignore) with the regular expressions

*.pyc
*~

Run svn propset on the project

svn -R propset svn:ignore -F .svnignore .

-R indicates that you are doing this recursively so all the directories ignore these.
-F indicates the file we just created with the regular expressions.

EDIT 5/27/08:
If you’ve already got yourself into the mess of versioning your compiled files, then you might want to do a few things.
1) Revert all the .pyc files (you can always recompile them). You don’t want there to be any local changes when you try to delete them:
find -name "*.pyc" -exec svn revert {} \;
2) Delete all the .pyc files using `svn delete`:
find -name "*.pyc" -exec svn delete {} \;
3) Finally recompile and re-run the svn ignore code above

Backing up with BackupPC

BackupPC is an awesome tool for backing up a number of different machines. BackupPC has a web application that manages your backups and a separate backup application that runs periodically based on the settings you have in the web admin.

The design is ingenious because it can back up a variety of machines with unique settings. For windows machines I use a protected share that can only be accessed by a user called ‘backup’. The backup application then connects periodically to the share and runs incremental backups. For the linux machines I run an rsync demon on the pc. This is one of the few cases where it’s as easy on windows as linux. (In case there’s any doubt… everything is easier on linux 99% of the time.)

Given that I use linux exclusively, here’s the documentation on setting up an rsyncd connection with Backuppc.

1) Install rsyncd: $ sudo apt-get install rsync (ubuntu)
2) Configure rsyncd with a backup option (/etc/rsyncd.conf)

[backup]
path = /home/jamstooks
exclude = "/home/jamstooks/.mozilla/firefox/***"
comment = Backup
uid = jamstooks
gid = jamstooks
read only = true
auth users = backup
secrets file = /etc/rsyncd.secrets

I excluded the firefox directory, because I was getting issues with the cache being accessed during backup. It would probably be best to include the bookmarks though.
/etc/rsyncd.secrets stores the passwords in plain-text, so make sure you adjust your permissions appropriately:

backup:passwd

3) Ubuntu sets up an init.d setup file, but you want to make sure the daemon is running…
4) Finally, you are ready to set up BackupPC. In the BackupPC admin, simply select rsync for this machine and fill in the password you selected.

More details can be found in the official BackupPC documentation.

Quick image resizing with python

I typically build automatic resizing into any django models that use ImageField by overriding the save function like so:

def save(self):
super(GalleryImage, self).save() # Call the "real" save() method
if self.image:
# now change the size of the image
path = settings.MEDIA_ROOT + self.image
img1 = PIL.open(path)
img2 = PIL.open(path)
img2.save(self.get_thumbnail_path())

size = 600,600
img1.thumbnail(size, PIL.ANTIALIAS)
img1.save(path)

# create the thumbnail
size = 150, 150
img2.thumbnail(size, PIL.ANTIALIAS)
img2.save(self.get_thumbnail_path())

def get_thumbnail_path(self):
path = settings.MEDIA_ROOT + self.image
return self.convert_path_to_thumbnail(path)

def get_thumbnail_url(self):
path = self.get_image_url()
return self.convert_path_to_thumbnail(path)

def convert_path_to_thumbnail(self, path):

basedir = os.path.dirname(path) + '/'
base, ext = os.path.splitext(os.path.basename(path))

# make thumbnail filename
th_name = base + "_tn"
th_name += ext
return urlparse.urljoin(basedir, th_name)

This ensures that all my images conform to the sizes I need. However, even if the server is automatically resizing the images for me, I don’t want to spend hours uploading huge images only to have them resized. It’s just a waste of bandwidth and time. So, I’ve written a quick python script that resizes all the images in a given directory and places the copies in a new directory called resized:


#! /usr/bin/env python

"""
What:
Resize all the jpg images in a directory
All images will be placed inside a directory called "resized"
This directory must exist
Usage: ./resize.py
"""

import PIL.Image as PIL
import re, os, sys, urlparse

SIZE = 600,600
JPG = re.compile(".*\.(jpg|jpeg)", re.IGNORECASE)

def get_new_path(path):
basedir = os.path.dirname(path) + '/resized/'
base, ext = os.path.splitext(os.path.basename(path))
file = base + ext

return urlparse.urljoin(basedir, file)

try:
image_dir = sys.argv[1]
except:
print "You must specify a directory"
exit(0)

files = os.listdir(image_dir)
for file in files:
if JPG.match(file):
f = image_dir.rstrip("/") + "/" + file
print f
img = PIL.open(f)
img.thumbnail(SIZE, PIL.ANTIALIAS)
img.save(get_new_path(f))

Server Uptime Monitoring with Python

We had a black out over the weekend, while I was away, so I decided to add a few more tools to keep me informed. See, when the power goes out the server can’t exactly let me know that it’s down, can it?

Fortunately, I have a slice at SliceHost which never goes down, so I decided to use that to monitor our in-house server. I put together a simple python script to do the job. It simply queries the HTTP Headers of the home page of each of our sites and emails me and sends a TXT to my cellphone if it gets anything other than a 200 status code.

The script itself is run by a cron demon every 15 minutes, but it won’t overwhelm my phone because I have a built-in Message Delay in the script.


#! /usr/bin/env python

import httplib
from urlparse import urlparse
from datetime import datetime, timedelta
import os, platform, time

# Modify your settings here
SITES = ( "http://www.site.com", ) # Sites to monitor
MAIL_FROM = "admin@site.com"
MAIL_TO = ["Twoway.*********@messaging.nextel.com","ben@site.com"]
MESSAGE_DELAY = 24 * 60 # The time between alerts in minutes
LOCK_FILE = "remote_mon_lock" # Lock file used for alert staggering

def send_alert(status, site):
"""
Send an email alert to MAIL_TO if the status is not 200
Use a Lock File to ensure that message don't get sent more
frequently than the MESSAGE_DELAY, to prevent large TXT bills.
"""
if status != 200:
if not os.path.exists(LOCK_FILE):
old_mod_date = datetime.now() - timedelta(minutes=MESSAGE_DELAY + 1)
old_timestamp = time.mktime(old_mod_date.timetuple())
f = open(LOCK_FILE, 'w')
f.close()
os.utime(LOCK_FILE, (old_timestamp, old_timestamp))

mod_date = datetime.fromtimestamp(os.path.getmtime(LOCK_FILE))
next_send_time = mod_date + timedelta(minutes=MESSAGE_DELAY)
if datetime.now() > next_send_time:
import smtplib
s = smtplib.SMTP()
s.connect()
s.sendmail(MAIL_FROM, MAIL_TO, "%s - %s" % (str(status), site))
s.close()
os.utime(LOCK_FILE, None)

for site in SITES:
url = urlparse(site)
error = ""
try:
conn = httplib.HTTPConnection(url[1])
# Use a HEAD request to get the status code
conn.request("HEAD", url[2])
status = conn.getresponse()
error = status.status
print "%d : %s : %s" % (status.status, site, datetime.now())
except:
print "Connection Failed : %s : %s" % (site, datetime.now())
error = "Connection Failed"

send_alert(error, site)

Nginx SSL passphrase at startup

I’m using nginx to serve all my media files and it is also a proxy to my apache server. I’ve been using SSL encryption for specific sections of the site, using nginx’s built in SSL support:

server {

listen 443;
server_name host.com;

access_log /var/log/nginx/register/host.access.log;

# SSL
ssl on;
ssl_certificate /home/jamstooks/ssl_certs/2008_cert/host.com.crt;
ssl_certificate_key /home/jamstooks/ssl_certs/2008_cert/host.com.key;

# Register
location ^~ /register/ {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;

proxy_set_header Host register.host.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

client_max_body_size 10m;
client_body_buffer_size 128k;

proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}

# Redirect everything else
location / {
rewrite ^(.*) http://www.host.com$1 permanent;
}

}

When I boot up Nginx it requests the passphrase for the encrypted certificate key. This is a huge problem though when there are unexpected shutdowns because the Nginx process won’t restart.

However, the problem is not with Nginx, but with the certificate itself. Because it is encrypted, Nginx can’t use it unless it until it has the pass-phrase. So, the easiest way to solve this is to provide Nginx with a decrypted version of the certificate key. The only issue is that you need to tie down the permissions on the file so that no one can access it at use it to impersonate you.

Apache details the process here:

# Remove the encryption from the RSA private key (while keeping a backup copy of the original file):

$ cp server.key server.key.org
$ openssl rsa -in server.key.org -out server.key

# Make sure the server.key file is only readable by root:

$ chmod 400 server.key

Less is More

I’m a firm believer that this saying holds true ALL THE TIME.

But in this case I’m just thinking about the `less` command. It’s just better than `more`.

So, if you’re using Linux and the bash shell put the following in your .bashrc file:

alias more='less'

ModelMultipleChoiceField save fails

I’ve been banging my head today trying to get a MultipleChoiceField or a ModelMultipleChoiceField to save correctly. I was finding that no matter which widgets I used, or which field types I was using, it was failing. I wouldn’t get any errors, and all the data would be saved except for my ManyToMany relationship.

So, after much ado creating custom widgets and scouring the web for advice, I found that I should have just read the ModelForm Documentation. There it was, under ‘The save() method”:

Another side effect of using commit=False is seen when your model has a many-to-many relation with another model. If your model has a many-to-many relation and you specify commit=False when you save a form, Django cannot immediately save the form data for the many-to-many relation. This is because it isn’t possible to save many-to-many data for an instance until the instance exists in the database.

I had been using save(commit=false) and hadn’t been using save_m2m(). That’ll teach me to read the documentation!