Technology for ponies

Dec 17, 2013

Admin interface with foreign key on a very long list

If your application has a model with a ForeignKey on an other model which gets a lot of entries in the database, the auto-generated admin interface can become a nightmare to load. Django will render your ForeignKey using a select drop-down and with thousands of entries loading time and browser memory are in bad shape.

Solution is rather simple, add your ForeignKey in the raw_id_fields parameter in the Django admin. Lets consider this example :


class MyModel(models.Model):
  user = models.ForeignKey(User)


from models.py import MyModel

class MyModelAdmin(admin.ModelAdmin):
    raw_id_fields = ("user",)

admin.site.register(MyModel, MyModelAdmin)

Sep 10, 2013

Django deployement : ubuntu, upstart, nginx, gunicorn and virtualenvwrapper


Previously my stack of choice for deploying django apps was apache + mod_wsgi. Recently I had to move the website you are currently reading to a new server. I took this opportunity to consider moving to a new stack. Here is my feedback and an in-depth guide on how to do it yourself.

I had often been told about ngnix, his low memory footprint and his high concurrency model so I decided to give it a try. I made a little benchmark, using apache AB, to test a high demanding django app I am working on. Without being a real breakthrough the benchmark turned in favor of nginx. So here we go :

The architecture of a stack based on nginx is different from apache + mod_wsgi : your code doesn't run embedded in the web server. ngnix acts as a reverse proxy : it receives incoming http connections, handles slow clients, SSL encryptions, caching, compressions, etc. and then pass the request on an application server. In the case of django apps the often mentioned solution is gunicorn. Gunicorn is a python WSGI HTTP Server. WSGI is a standardized interface between Web Servers (like ngnix) and application servers (like gunicorn).

Introduction is over, here is how to configure all those things to work together. The underlying OS is ubuntu so your mileage may vary if you are using an other linux distribution, especially on the init scripts, that are written for upstart.


Installation :

# apt-get install nginx

Create a new virtual host for ngnix in /etc/nginx/sites-available/ponytech:

server {
  listen 80;
  server_name ponytech.net;
  root /home/deploy/ponytech;

  access_log /var/log/nginx/ponytech.log;

  location /static {
    alias /home/deploy/ponytech/collectstatic;

  location / {
    proxy_pass_header Server;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Scheme $scheme;
    proxy_connect_timeout 6000;
    proxy_read_timeout 6000;
    proxy_pass http://localhost:8000/;

Please note :

  • collectstatic is name of the folder I configured to hold all the static files from my apps. /static is the URL I configured to serve those files. Please adjust them to what your STATIC_ROOT and STATIC_URL settings points to.
  • http://localhost:8000 is the ip and port where the request will be passed on, we will need it later to configure gunicorn
  • to understand all the other settings, please refer to nginx documentation

Enable the new virtual host with :

# ln -s /etc/nginx/sites-available/ponytech /etc/nginx/sites-enabled/ponytech

# service nginx reload

Deploy your code

I like to create a new user on the system that holds the application files and run the application server:

# useradd deploy -m -d /home/www -s /bin/bash

Now switch user to deploy and get your project source using your usual VCS (here with git) :

# su - deploy

$ git clone git@mygitserver:/ponytech.git

Next step is to install the dependencies of your project. For this purpose we use virtualenv and a very handy extension on top of it : virtualenvwrapper. Virtualenv allows one to have different versions of the same dependencies both installed on your system. It is a very good practice to use it, and it turns out to be mandatory if you want to run at the same time your old project that rely on Django 1.3, and your brand new one on Django 1.5.

Let's install and configure virtualenvwrapper :

$ pip install virtualenvwrapper

$ mkdir ~/.virtualenvs ~/.pip_packages

Add to the end of your .bashrc file :

export WORKON_HOME=$HOME/.virtualenvs

export PIP_DOWNLOAD_CACHE=$HOME/.pip_packages


source /usr/local/bin/virtualenvwrapper.sh

And source it:

$ source ~/.bashrc

The PIP_DOWNLOAD_CACHE environnement variable is not related to virtualenv but it makes pip cache and not download again packages you share between virtual environnements.

Now let's create our virtualenv and install its dependencies:

$ mkvirtualenv ponytech

$ pip install -r requirements.txt

$ pip install gunicorn

Configure the startup script

Create the init script for upstart in /etc/init/ponytech.conf :

description "Ponytech website"
start on runlevel [2345]
stop on runlevel [06]
respawn limit 10 5

                LOGDIR=$(dirname $LOGFILE)
                test -d $LOGDIR || mkdir -p $LOGDIR
                cd /home/deploy/$NAME
                exec /home/deploy/.virtualenvs/$NAME/bin/gunicorn_django \
                         -w $NUM_WORKERS -t $TIMEOUT \
                        --user=$USER --group=$GROUP --log-level=debug \
                        --name=$NAME -b$PORT \
                        --log-file=$LOGFILE 2>>$LOGFILE
end script

Note that the PORT=8000 must match the number you configured in nginx configuration. In case you are running multiple websites on the same machine you'll have to increment this number accordingly.

Add new system service:

# ln -fs /lib/init/upstart-job /etc/init.d/ponytech

Make it starts at system boot:

# update-rc.d ponytech defaults

And start it now :

# service ponytech start


Voila! We now have our new web stack configured and running. It appears a bit more tedious than apache + mod_wsgi to get it up but it prove to be far more powerful and scalable. Nginx and gunicorn can run on different machines, you can even easily set up load balancing with several instances of it.

The procedure described here is how I set up my own server, don't take it as an immutable reference as there are for sure many other ways of doing it. Please leave a comment to suggest any improvements!

Apr 29, 2013

Save a model without triggering a signal nor the save() method

It happens you may want to save one of your model but without triggering any pre_save / post_save signal, nor the save() method of your model. Reasons could be, your are restoring some fixtures data, you don't want to update the object modification date that is configured with auto_now = True, you want to skip some validations, etc.

There had been a ticket opened in the past to request such a feature for Django but this request had been rejected.

So here is a quick tip to allow you to achieve this behavior if you really have to : you can use django bulk update feature. Because this function only performs a raw SQL query its does not actually knows what object you are updating no signals and no save() methods are triggered / called.

Let say you have the following model with a DateTimeField, auto_now set to True:

class Document(models.Model)
   title = models.CharField(max_length=255)
   last_modified = models.DateTimeField(auto_now=True)

To update the title of a Document without updating his last_modified date you can do:

doc = Document.objects.get(title='Old title')
Document.objects.filter(id=doc.id).update(title='New title')
← Previous Next → Page 2 of 3