Ponytech

Introduction

Previously my stack of choice for deploying django apps was apache + mod_wsgi. Recently I had to move the website you are currently reading to a new server. I took this opportunity to consider moving to a new stack. Here is my feedback and an in-depth guide on how to do it yourself.

I had often been told about ngnix, his low memory footprint and his high concurrency model so I decided to give it a try. I made a little benchmark, using apache AB, to test a high demanding django app I am working on. Without being a real breakthrough the benchmark turned in favor of nginx. So here we go :

The architecture of a stack based on nginx is different from apache + mod_wsgi : your code doesn't run embedded in the web server. ngnix acts as a reverse proxy : it receives incoming http connections, handles slow clients, SSL encryptions, caching, compressions, etc. and then pass the request on an application server. In the case of django apps the often mentioned solution is gunicorn. Gunicorn is a python WSGI HTTP Server. WSGI is a standardized interface between Web Servers (like ngnix) and application servers (like gunicorn).

Introduction is over, here is how to configure all those things to work together. The underlying OS is ubuntu so your mileage may vary if you are using an other linux distribution, especially on the init scripts, that are written for upstart.

Webserver

Installation :

# apt-get install nginx

Create a new virtual host for ngnix in /etc/nginx/sites-available/ponytech:

server {
  listen 80;
  server_name ponytech.net;
  root /home/deploy/ponytech;

  access_log /var/log/nginx/ponytech.log;

  location /static {
    alias /home/deploy/ponytech/collectstatic;
  }

  location / {
    proxy_pass_header Server;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Scheme $scheme;
    proxy_connect_timeout 6000;
    proxy_read_timeout 6000;
    proxy_pass http://localhost:8000/;
  }
}

Please note :

• collectstatic is name of the folder I configured to hold all the static files from my apps. /static is the URL I configured to serve those files. Please adjust them to what your STATIC_ROOT and STATIC_URL settings points to.
• http://localhost:8000 is the ip and port where the request will be passed on, we will need it later to configure gunicorn
• to understand all the other settings, please refer to nginx documentation

Enable the new virtual host with :

# ln -s /etc/nginx/sites-available/ponytech /etc/nginx/sites-enabled/ponytech

# service nginx reload

Deploy your code

I like to create a new user on the system that holds the application files and run the application server:

# useradd deploy -m -d /home/www -s /bin/bash

Now switch user to deploy and get your project source using your usual VCS (here with git) :

# su - deploy

$ git clone git@mygitserver:/ponytech.git

Next step is to install the dependencies of your project. For this purpose we use virtualenv and a very handy extension on top of it : virtualenvwrapper. Virtualenv allows one to have different versions of the same dependencies both installed on your system. It is a very good practice to use it, and it turns out to be mandatory if you want to run at the same time your old project that rely on Django 1.3, and your brand new one on Django 1.5.

Let's install and configure virtualenvwrapper :

$ pip install virtualenvwrapper

$ mkdir ~/.virtualenvs ~/.pip_packages

Add to the end of your .bashrc file :

export WORKON_HOME=$HOME/.virtualenvs

export PIP_DOWNLOAD_CACHE=$HOME/.pip_packages

export PROJECT_HOME=$HOME/

source /usr/local/bin/virtualenvwrapper.sh

And source it:

$ source ~/.bashrc

The PIP_DOWNLOAD_CACHE environnement variable is not related to virtualenv but it makes pip cache and not download again packages you share between virtual environnements.

Now let's create our virtualenv and install its dependencies:

$ mkvirtualenv ponytech

$ pip install -r requirements.txt

$ pip install gunicorn

Configure the startup script

Create the init script for upstart in /etc/init/ponytech.conf :

description "Ponytech website"
start on runlevel [2345]
stop on runlevel [06]
respawn
respawn limit 10 5

script
                NAME=ponytech
                PORT=8000
                NUM_WORKERS=2
                TIMEOUT=120
                USER=deploy
                GROUP=deploy
                LOGFILE=/var/log/gunicorn/$NAME.log
                LOGDIR=$(dirname $LOGFILE)
                test -d $LOGDIR || mkdir -p $LOGDIR
                cd /home/deploy/$NAME
                exec /home/deploy/.virtualenvs/$NAME/bin/gunicorn_django \
                         -w $NUM_WORKERS -t $TIMEOUT \
                        --user=$USER --group=$GROUP --log-level=debug \
                        --name=$NAME -b 127.0.0.1:$PORT \
                        --log-file=$LOGFILE 2>>$LOGFILE
end script

Note that the PORT=8000 must match the number you configured in nginx configuration. In case you are running multiple websites on the same machine you'll have to increment this number accordingly.

Add new system service:

# ln -fs /lib/init/upstart-job /etc/init.d/ponytech

Make it starts at system boot:

# update-rc.d ponytech defaults

And start it now :

# service ponytech start

Conclusion

Voila! We now have our new web stack configured and running. It appears a bit more tedious than apache + mod_wsgi to get it up but it prove to be far more powerful and scalable. Nginx and gunicorn can run on different machines, you can even easily set up load balancing with several instances of it.

The procedure described here is how I set up my own server, don't take it as an immutable reference as there are for sure many other ways of doing it. Please leave a comment to suggest any improvements!